-- Logs begin at Mon 2022-08-29 21:50:30 UTC, end at Mon 2022-08-29 23:01:39 UTC. -- Aug 29 22:25:41.052054 dv-cinder-854270-4-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 29 22:25:41.060174 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: *** Starting uWSGI 2.0.18-debian (64bit) on [Mon Aug 29 22:25:41 2022] *** Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: nodename: dv-cinder-854270-4-iscsi Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: machine: x86_64 Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: clock source: unix Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: pcre jit disabled Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: detected number of CPU cores: 4 Aug 29 22:25:41.062220 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: current working directory: / Aug 29 22:25:41.063042 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: detected binary path: /usr/bin/uwsgi-core Aug 29 22:25:41.063042 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: your processes number limit is 47680 Aug 29 22:25:41.063042 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: your memory page size is 4096 bytes Aug 29 22:25:41.063042 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: detected max file descriptor number: 2048 Aug 29 22:25:41.063042 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: lock engine: pthread robust mutexes Aug 29 22:25:41.063042 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: thunder lock: enabled Aug 29 22:25:41.063042 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 29 22:25:41.063042 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 29 22:25:41.104350 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: Python main interpreter initialized at 0x5565f99d09c0 Aug 29 22:25:41.104350 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: python threads support enabled Aug 29 22:25:41.104350 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: your server socket listen backlog is limited to 100 connections Aug 29 22:25:41.104350 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: your mercy for graceful operations on workers is 90 seconds Aug 29 22:25:41.104691 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: mapped 403077 bytes (393 KB) for 2 cores Aug 29 22:25:41.104767 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: *** Operational MODE: preforking *** Aug 29 22:25:41.105028 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: *** uWSGI is running in multiple interpreter mode *** Aug 29 22:25:41.105028 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: spawned uWSGI master process (pid: 102992) Aug 29 22:25:41.105219 dv-cinder-854270-4-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 29 22:25:41.107124 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: spawned uWSGI worker 1 (pid: 103002, cores: 1) Aug 29 22:25:41.107124 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: spawned uWSGI worker 2 (pid: 103003, cores: 1) Aug 29 22:25:41.107570 dv-cinder-854270-4-iscsi devstack@keystone.service[102992]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 29 22:25:45.431099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 29 22:25:45.431099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: from cryptography.utils import int_from_bytes Aug 29 22:25:45.431901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 29 22:25:45.431901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: from cryptography.utils import int_from_bytes Aug 29 22:25:45.432378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 29 22:25:45.432378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: from cryptography.utils import int_from_bytes Aug 29 22:25:45.432487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 29 22:25:45.432487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: from cryptography.utils import int_from_bytes Aug 29 22:25:45.482400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 29 22:25:45.482606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 29 22:25:45.482705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 29 22:25:45.482820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 29 22:25:45.482888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] command line args: [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 29 22:25:45.482954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] command line args: [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 29 22:25:45.483067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 29 22:25:45.483205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 29 22:25:45.483291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ================================================================================ {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 29 22:25:45.483377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ================================================================================ {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 29 22:25:45.483489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] admin_token = **** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.483582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] admin_token = **** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.483739 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] config_dir = [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.483840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] config_dir = [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.483921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.483994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] config_source = [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] config_source = [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] debug = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] debug = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.484994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] insecure_debug = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] insecure_debug = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] list_limit = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] list_limit = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] log_config_append = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.485964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] log_config_append = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] log_dir = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] log_dir = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] log_file = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] log_file = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.486891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.487917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488729 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.488926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.489007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.489148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.489229 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.489372 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] notification_format = cadf {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.489453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] notification_format = cadf {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.489643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.489758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.490129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.77/identity {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.490209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.77/identity {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.490336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] publish_errors = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.490411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] publish_errors = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.491168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.491245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.491357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.491480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.491616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.491696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.491862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.491963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] standard_threads = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] standard_threads = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] strict_password_check = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] strict_password_check = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] use_eventlog = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.492989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] use_journal = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] use_eventlog = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] use_json = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] use_journal = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] use_stderr = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] use_json = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] use_syslog = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] use_stderr = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493745 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] use_syslog = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] watch_log_file = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.493979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] watch_log_file = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 29 22:25:45.494063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.494233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.494312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.494472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.494574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.494771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.494865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.494998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.495111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.495231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.495326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.495474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.495563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.495724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.495819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.495959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.496049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.496199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.496284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.496443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.496521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.496674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.496770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.496959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.497921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.498087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.498166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.498333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.498412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.498575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.498661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.498839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.498920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.499076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.499156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.499328 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.499411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.499576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.499662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.499821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.499900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.500968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.501972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.502132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.502327 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.502411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.502473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.502594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.502670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.502836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.502940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.503952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.504969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.505992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.506128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.506875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.507955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.508996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.509085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.509234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.509324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.509459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.509546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.509715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.509796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.509952 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.510041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.510185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.510266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.510422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.510503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.510658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.510773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.510932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511487 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.511945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.512097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.512178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.enabled = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.512332 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.512412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.512562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.512643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.enabled = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.512801 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.512888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513739 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.513972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.514061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.514200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.514280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.514432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.514512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.514664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.514871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.514952 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.515971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.516118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.516205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.516349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.516440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.516581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.516661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.516808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.516887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517036 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.517939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.518978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.519136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.519229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.519379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.519465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.519610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] auth.external = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.519717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.519839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] auth.mapped = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.519922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.520084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] auth.external = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.520173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.520316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] auth.mapped = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.520397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.520562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] auth.password = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.520641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.520787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] auth.token = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.520870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.521024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.521105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] auth.password = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.521256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] catalog.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.521336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] auth.token = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.521497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.521577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.521759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] catalog.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.521844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.522010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.522108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.522321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.522442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.522530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.522651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.522752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] credential.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.523185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.523413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] credential.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.523499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.523580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.523688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.523771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] credential.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.523929 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] credential.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.524976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.525127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.525210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.525373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.525466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.525630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.525713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.525869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.525950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.526124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.526218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.526362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.526444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.526615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.526812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.526893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] federation.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.526980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.527094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.527183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.527337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] federation.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.527419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.527579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.527669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] federation.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.527807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.527889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.528051 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.528139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] federation.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.528285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.528379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.528534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.528615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.528769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.528850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.529947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.530029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.530191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.530272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.530424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.530505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.530655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.530757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.530923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.531049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.531145 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.531222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.531378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.531461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.531625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.531802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.531907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.532928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.533042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.533171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.533261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.533415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.533498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.533667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.533748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.533910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534001 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.534956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.535123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.535205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.535351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.535442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.535584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.535666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.535822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.535904 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.536996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.537076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.537228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.537327 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.537502 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.537596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.537737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.537819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.537983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.538065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.538216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.538307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.538442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.password = **** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.538524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.538676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.538805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.538932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.password = **** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.539953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.540110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.540191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.540336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.540425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.540563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.540645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.540806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.540887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.541953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.542036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.542189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.542270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.542417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.542500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.542657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.542769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.542922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.543932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.544993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.545073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.545229 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.545311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.545466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.545546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.545705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.545786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.545945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.546986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.547138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.547220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.547368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.547452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.547601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.547688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.547836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.547918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.548069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.548150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.548314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.548394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] policy.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.548538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.548618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.548782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.548872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] policy.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] receipt.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] receipt.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.549938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.550019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.550167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.550309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.550408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.550589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.550708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] resource.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.550843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.550940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.551086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.551167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] resource.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.551330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.551417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] resource.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.551575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.551737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.551814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] resource.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.551885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.552042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] revoke.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.552121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.552285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.552373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.552523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.552630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.552754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] role.cache_time = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.552824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] revoke.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.553533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] role.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.553617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.553778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] role.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.553858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.554007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] role.list_limit = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.554325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] role.cache_time = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.554436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.554515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.554576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] role.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.554711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.554811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] role.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.554975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.555074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] role.list_limit = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.555223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.555340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.555540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.555630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.556934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.557056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.557148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.557387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.557468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.557636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.557718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.557882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.557973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.558128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.558210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.558391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.558551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.558632 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.558809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.558897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.559062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.559145 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.559314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.559410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.559564 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.559647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.559798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.559932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.560065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.560206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.560336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.560476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.560562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.560693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.560825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.560964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.561093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.561223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.561351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.561488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.561616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.561751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.561881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] token.cache_time = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.562009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.562147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] token.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.562276 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.562403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.562541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.562670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] token.provider = fernet {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.562816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.562948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.563077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.563204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.563343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] token.cache_time = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.563472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.563599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] token.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.563727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.563855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.563992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.564129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] token.provider = fernet {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.564257 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.564384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.564512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] trust.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.564641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.564768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.564902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.565031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.565160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.565287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.565413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.565541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.565677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.565817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.565945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] trust.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.566073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.566210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.566346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.566475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.566614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.566775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.566901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.connection = **** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.567031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.567162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.567290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.567429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.567570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.567707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.567845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.567974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.568101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.568231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.568358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.connection = **** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.568499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.568672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.569317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.569448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.569575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.569701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.569795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.569931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.570058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.570143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.570253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.570386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.570524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.570653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.570810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.570936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=103002) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 29 22:25:45.571924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.571924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.572258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 29 22:25:45.572258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=103003) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 29 22:25:45.600327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.600591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.600693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.600817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.600906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.601112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.609326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.609519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.639064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.639265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.639265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.639487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.696136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.696318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.746142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.746375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.747327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.747565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.904824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.905955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.906075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.906287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.930379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.930611 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.930708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.930863 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.930980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931276 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.931924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.943048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.943377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.956419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.956568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.956666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.956746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.956822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.956897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.957003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.957090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.957217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.957305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.957430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.957516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.957642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=103002) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.957738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=103003) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 29 22:25:45.974633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.975400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 29 22:25:45.975496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.975709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 29 22:25:45.975816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.975938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 29 22:25:45.976050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.976213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.976298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 29 22:25:45.976476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.976571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.976737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.976820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.976996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.977078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.977519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.986858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.986987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.987152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.987240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.987427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.987533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.987713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.987800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.995378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:45.995543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:45.998292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:45.998449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:45.998583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.998766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.998901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.999045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.999189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.999337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.999476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.999621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.999739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:45.999884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.000011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.000156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.000286 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.000430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.000561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.000818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.000902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.001093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.001175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.001362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.024691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.024872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.026191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.026296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.026479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.026585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.031087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.031235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.031360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.031512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.031647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.031794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.038802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.038945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.039079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.039188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.043100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.043378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.043466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.043665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.043748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.043951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.044035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.044230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.044314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.044514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.044797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.045068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.045344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.046396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.046689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.047000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.063903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.064188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.064728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.065024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.068065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.068904 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.071480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.071777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.072123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.072326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.076008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.076649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.079157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.079774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.082389 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.082700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.082857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.083012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.083155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.083346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.083443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.083643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.083727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.083921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.084097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.084914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.101402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.101548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.101695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.101843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.101981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.102113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.102223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.102378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.109068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.109305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.110758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.110871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.112314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.112399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.112607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.112691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.115313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.115406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.115631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.115727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.115918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.116113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.116291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.116439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.116565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.116666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.127914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.128054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.128196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.128287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.128483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.128582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.128765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.128881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.129047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.129144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.129329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.129425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.129616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.129714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.145210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.145543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.149103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.149261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.152033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.152171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.153497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.153630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.155045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.155178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.155339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.155511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.155635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.155765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.162112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.162304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.164291 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.164499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.167234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.167409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.167539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.167701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.167842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.167969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.168100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.168232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.176087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.176256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.176399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.176538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.176670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.176799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.183629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=103003) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.183817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=103002) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 29 22:25:46.183997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.184156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.184289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.184412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.184547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.184701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.184893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.185041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.185224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.185387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.185515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.185675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.185807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.185933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.186067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.186204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.186330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.186462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.186592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.186753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.186883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.187009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.187135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.187261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.187396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=103003) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.187523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=103002) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 29 22:25:46.221781 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5565f99d09c0 pid: 103003 (default app) Aug 29 22:25:46.222014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5565f99d09c0 pid: 103002 (default app) Aug 29 22:25:46.225426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e2a6f-1b0a-40f8-b78f-f695a83b57d4 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:46.225716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e2a6f-1b0a-40f8-b78f-f695a83b57d4 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:46.225993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e2a6f-1b0a-40f8-b78f-f695a83b57d4 None None] PATH_INFO: `/v3` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:46.231356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1/1] 10.222.0.77 () {50 vars in 791 bytes} [Mon Aug 29 22:25:46 2022] GET /identity/v3/ => generated 255 bytes in 9 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:25:52.995533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d9802-f339-4f56-83f0-520ae5b76acd None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:52.996264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d9802-f339-4f56-83f0-520ae5b76acd None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:52.996264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d9802-f339-4f56-83f0-520ae5b76acd None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:53.000362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1/2] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:25:52 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:25:53.006010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:53.006653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:53.006653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:53.078311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: INFO dbcounter [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] Registered counter for database keystone Aug 29 22:25:53.098056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG oslo_db.sqlalchemy.engines [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=103003) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 29 22:25:53.106309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writer thread running {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 29 22:25:53.296788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG passlib.handlers.bcrypt [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=103003) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 29 22:25:53.297186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG passlib.handlers.bcrypt [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=103003) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 29 22:25:53.357055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:25:53.358604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:25:53.493535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 2/3] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:25:53 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 488 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Aug 29 22:25:53.958275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1d9802-f339-4f56-83f0-520ae5b76acd None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:54.029525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: INFO dbcounter [None req-fb1d9802-f339-4f56-83f0-520ae5b76acd None None] Registered counter for database keystone Aug 29 22:25:54.051732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG oslo_db.sqlalchemy.engines [None req-fb1d9802-f339-4f56-83f0-520ae5b76acd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=103002) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 29 22:25:54.058918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writer thread running {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 29 22:25:54.186757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1d9802-f339-4f56-83f0-520ae5b76acd None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:54.191176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20842f18-12d2-43c3-8327-f18f3c5a6cb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:54.191816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20842f18-12d2-43c3-8327-f18f3c5a6cb8 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:54.192097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20842f18-12d2-43c3-8327-f18f3c5a6cb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:54.192374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20842f18-12d2-43c3-8327-f18f3c5a6cb8 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:54.204587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20842f18-12d2-43c3-8327-f18f3c5a6cb8 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:54.258413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING keystone.common.rbac_enforcer.enforcer [None req-20842f18-12d2-43c3-8327-f18f3c5a6cb8 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 29 22:25:54.258766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20842f18-12d2-43c3-8327-f18f3c5a6cb8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:25:54.269452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 2/4] 10.222.0.77 () {58 vars in 1324 bytes} [Mon Aug 29 22:25:53 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 29 22:25:54.275904 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:54.290943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7cf142-57d5-4ca4-99d2-d155fd49ed12 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:54.295417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c8d7ca-0f97-4aba-85c4-4ea899951733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:54.296554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c8d7ca-0f97-4aba-85c4-4ea899951733 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:54.296835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c8d7ca-0f97-4aba-85c4-4ea899951733 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:54.297107 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c8d7ca-0f97-4aba-85c4-4ea899951733 admin admin] PATH_INFO: `/v3/projects/fabab14479bd4d4c870f8f76efa1b499` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:54.298691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c8d7ca-0f97-4aba-85c4-4ea899951733 admin admin] RBAC: Authorizing `identity:get_project(project_id=fabab14479bd4d4c870f8f76efa1b499)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:54.360676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING keystone.common.rbac_enforcer.enforcer [None req-97c8d7ca-0f97-4aba-85c4-4ea899951733 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 29 22:25:54.361032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c8d7ca-0f97-4aba-85c4-4ea899951733 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:25:54.365779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 3/5] 10.222.0.77 () {56 vars in 1207 bytes} [Mon Aug 29 22:25:54 2022] GET /identity/v3/projects/fabab14479bd4d4c870f8f76efa1b499 => generated 344 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:25:56.272090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73ae639-a45d-4e72-b74c-650e6fcea122 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:56.272656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73ae639-a45d-4e72-b74c-650e6fcea122 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:56.272656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73ae639-a45d-4e72-b74c-650e6fcea122 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:56.277139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 3/6] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:25:56 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:25:56.282528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d200d253-72d4-4811-9617-b265751d8c6a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:56.282956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d200d253-72d4-4811-9617-b265751d8c6a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:56.283115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d200d253-72d4-4811-9617-b265751d8c6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:56.304536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d200d253-72d4-4811-9617-b265751d8c6a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:25:56.306096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d200d253-72d4-4811-9617-b265751d8c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:25:56.314361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 4/7] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:25:56 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 29 22:25:56.584153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73ae639-a45d-4e72-b74c-650e6fcea122 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:56.596973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73ae639-a45d-4e72-b74c-650e6fcea122 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:56.601913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f4ad97-35c4-4143-8aa6-c63390ae3537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:56.602516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f4ad97-35c4-4143-8aa6-c63390ae3537 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:56.602827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f4ad97-35c4-4143-8aa6-c63390ae3537 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:56.603104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f4ad97-35c4-4143-8aa6-c63390ae3537 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:56.611519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f4ad97-35c4-4143-8aa6-c63390ae3537 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:56.613311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f4ad97-35c4-4143-8aa6-c63390ae3537 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:25:56.622512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 4/8] 10.222.0.77 () {58 vars in 1324 bytes} [Mon Aug 29 22:25:56 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 29 22:25:56.628398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d200d253-72d4-4811-9617-b265751d8c6a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:56.637849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d200d253-72d4-4811-9617-b265751d8c6a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:56.642321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca954044-9a81-4a42-a7d9-673b505d9fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:56.642983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca954044-9a81-4a42-a7d9-673b505d9fd7 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:56.643263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca954044-9a81-4a42-a7d9-673b505d9fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:56.643528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca954044-9a81-4a42-a7d9-673b505d9fd7 admin admin] PATH_INFO: `/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:56.644872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca954044-9a81-4a42-a7d9-673b505d9fd7 admin admin] RBAC: Authorizing `identity:get_user(user_id=2f1e799f7d5b4182a2e0ecd338aaa415)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:56.646701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca954044-9a81-4a42-a7d9-673b505d9fd7 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:25:56.648777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 5/9] 10.222.0.77 () {56 vars in 1198 bytes} [Mon Aug 29 22:25:56 2022] GET /identity/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:25:58.729109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e2b46b-aae1-4402-8b66-d0d2c3620617 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:58.729696 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e2b46b-aae1-4402-8b66-d0d2c3620617 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:58.730052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e2b46b-aae1-4402-8b66-d0d2c3620617 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:58.732383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 5/10] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:25:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:25:58.740513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5753358-0c76-4bed-9403-ae0ec4c5c3f7 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:58.741030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5753358-0c76-4bed-9403-ae0ec4c5c3f7 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:58.741190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5753358-0c76-4bed-9403-ae0ec4c5c3f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:58.761966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b5753358-0c76-4bed-9403-ae0ec4c5c3f7 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:25:58.763351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b5753358-0c76-4bed-9403-ae0ec4c5c3f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:25:58.775057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166a2ca7-feb6-4e8c-9e0a-ab5bd00469f5 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:58.775468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166a2ca7-feb6-4e8c-9e0a-ab5bd00469f5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:58.775877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166a2ca7-feb6-4e8c-9e0a-ab5bd00469f5 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:58.777358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 6/11] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:25:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:25:58.783175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:58.783467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:58.783713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:58.828989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG passlib.handlers.bcrypt [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=103002) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 29 22:25:58.829369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG passlib.handlers.bcrypt [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=103002) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 29 22:25:58.865535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 6/12] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:25:58.879254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:25:58.880870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:25:59.000921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 7/13] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:25:59.142905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5753358-0c76-4bed-9403-ae0ec4c5c3f7 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:59.178058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:25:59.273081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5753358-0c76-4bed-9403-ae0ec4c5c3f7 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:59.317023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892b13e4-c4fd-4a84-848d-97a006cb230b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:59.317650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892b13e4-c4fd-4a84-848d-97a006cb230b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:59.317937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892b13e4-c4fd-4a84-848d-97a006cb230b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:59.318214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892b13e4-c4fd-4a84-848d-97a006cb230b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:59.333542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:59.402544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892b13e4-c4fd-4a84-848d-97a006cb230b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:59.404302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892b13e4-c4fd-4a84-848d-97a006cb230b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:25:59.433291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:25:59.459034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a179097b-8404-4599-b05d-b29ea8d701c2 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:59.462146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:25:59.516258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baa0d4d-9b6a-4591-a985-09d31a5ae60e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:59.517552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa0d4d-9b6a-4591-a985-09d31a5ae60e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:59.518017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa0d4d-9b6a-4591-a985-09d31a5ae60e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:59.518594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa0d4d-9b6a-4591-a985-09d31a5ae60e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:59.540446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 7/14] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:25:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:25:59.543776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5baa0d4d-9b6a-4591-a985-09d31a5ae60e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:25:59.546843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892b13e4-c4fd-4a84-848d-97a006cb230b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5baa0d4d-9b6a-4591-a985-09d31a5ae60e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:25:59.557955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:25:59.559828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:25:59.561317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 8/15] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:25:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:25:59.565317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baa0d4d-9b6a-4591-a985-09d31a5ae60e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:59.681037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892b13e4-c4fd-4a84-848d-97a006cb230b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:59.702943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baa0d4d-9b6a-4591-a985-09d31a5ae60e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:59.719633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:25:59.736979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:59.737982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:59.738434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:59.738793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:59.739237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:59.739415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:59.739707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:59.739975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:59.740656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:25:59.740962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:59.743050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:25:59.755810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 9/16] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:25:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:25:59.758213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:59.760960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:25:59.762373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:25:59.781650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:25:59.783589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:25:59.785243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:25:59.785243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:25:59.785243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:25:59.785243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 8/17] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:25:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:25:59.789372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:59.798285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:25:59.891595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cee7a7-0dcb-45ea-b5c8-0e53f93607df None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:59.933794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33d6c74-90f0-4144-a0c2-eb0d59cbf496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:59.934621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d6c74-90f0-4144-a0c2-eb0d59cbf496 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:59.935027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d6c74-90f0-4144-a0c2-eb0d59cbf496 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:59.935305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d6c74-90f0-4144-a0c2-eb0d59cbf496 None admin] PATH_INFO: `/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:59.937079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33d6c74-90f0-4144-a0c2-eb0d59cbf496 None admin] RBAC: Authorizing `identity:get_user(user_id=2f1e799f7d5b4182a2e0ecd338aaa415)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:59.938890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33d6c74-90f0-4144-a0c2-eb0d59cbf496 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:25:59.940924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 10/18] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:25:59 2022] GET /identity/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415 => generated 247 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:25:59.941087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af6f774-1845-401b-a780-04b135a07d42 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:25:59.949925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33d6c74-90f0-4144-a0c2-eb0d59cbf496 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:25:59.992404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:25:59.993120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:25:59.993408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:25:59.993683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:25:59.994612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:25:59.994955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:25:59.996584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:00.015568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 9/19] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:25:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:00.083499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:00.091801 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33d6c74-90f0-4144-a0c2-eb0d59cbf496 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:00.137704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e04215b-ed44-45c1-bbf0-0d0f42436db6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:00.138365 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04215b-ed44-45c1-bbf0-0d0f42436db6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:00.138640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04215b-ed44-45c1-bbf0-0d0f42436db6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:00.138947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04215b-ed44-45c1-bbf0-0d0f42436db6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:00.140284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e04215b-ed44-45c1-bbf0-0d0f42436db6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:00.141820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e04215b-ed44-45c1-bbf0-0d0f42436db6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:00.143619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 11/20] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:25:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:00.150281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:00.197074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:00.290482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327e815c-c9d5-48c8-95c7-b08170806f60 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:00.344109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38213e3-2202-4c5b-8437-991dfef575dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:00.345088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38213e3-2202-4c5b-8437-991dfef575dc None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:00.345573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38213e3-2202-4c5b-8437-991dfef575dc None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:00.346081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38213e3-2202-4c5b-8437-991dfef575dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:00.346875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38213e3-2202-4c5b-8437-991dfef575dc None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.domain.id=default, user.id=2f1e799f7d5b4182a2e0ecd338aaa415) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:00.347199 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38213e3-2202-4c5b-8437-991dfef575dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:00.348672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38213e3-2202-4c5b-8437-991dfef575dc None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:00.367261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 10/21] 10.222.0.77 () {56 vars in 1322 bytes} [Mon Aug 29 22:26:00 2022] GET /identity/v3/role_assignments?user.id=2f1e799f7d5b4182a2e0ecd338aaa415&scope.domain.id=default&role.id=38171c89968943bd92d9e9758db65615 => generated 233 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:02.699137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4965a3-ace6-4c48-8c18-de3f985cdcd6 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:02.700649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4965a3-ace6-4c48-8c18-de3f985cdcd6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:02.701385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4965a3-ace6-4c48-8c18-de3f985cdcd6 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:02.703685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 12/22] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:02 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:02.712115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c84c4a0-0b0d-44fc-bd52-963ccc9bdcd6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:02.713172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c84c4a0-0b0d-44fc-bd52-963ccc9bdcd6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:02.713682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c84c4a0-0b0d-44fc-bd52-963ccc9bdcd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:02.760160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2c84c4a0-0b0d-44fc-bd52-963ccc9bdcd6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:02.761688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2c84c4a0-0b0d-44fc-bd52-963ccc9bdcd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:02.888556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 11/23] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:02.956280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c34fe-c274-4fac-90cc-3a2f5b81d0a8 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:02.956663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c34fe-c274-4fac-90cc-3a2f5b81d0a8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:02.957093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9c34fe-c274-4fac-90cc-3a2f5b81d0a8 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:02.959306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 13/24] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:02.966124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa0866-9bfc-40d3-ab01-4cb3127bbaf8 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:02.966448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa0866-9bfc-40d3-ab01-4cb3127bbaf8 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:02.966695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa0866-9bfc-40d3-ab01-4cb3127bbaf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:02.992200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-63fa0866-9bfc-40d3-ab01-4cb3127bbaf8 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:02.993885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-63fa0866-9bfc-40d3-ab01-4cb3127bbaf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:03.035262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:03.070820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197cb5fd-23f0-402f-9142-753be5cb97ea None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:03.071212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197cb5fd-23f0-402f-9142-753be5cb97ea None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:03.071425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197cb5fd-23f0-402f-9142-753be5cb97ea None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:03.072927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 14/25] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:03.082402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15c98e0-1651-4109-a3b1-d6fcbc3d7ba3 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:03.082800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15c98e0-1651-4109-a3b1-d6fcbc3d7ba3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:03.083012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15c98e0-1651-4109-a3b1-d6fcbc3d7ba3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:03.118888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c15c98e0-1651-4109-a3b1-d6fcbc3d7ba3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:03.120872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c15c98e0-1651-4109-a3b1-d6fcbc3d7ba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:03.124224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 12/26] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:03.254912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 15/27] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Aug 29 22:26:03.343004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fa0866-9bfc-40d3-ab01-4cb3127bbaf8 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:03.503165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fa0866-9bfc-40d3-ab01-4cb3127bbaf8 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:03.570725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:03.571785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:03.572299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:03.572770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:03.594954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:03.596788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:03.609529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15c98e0-1651-4109-a3b1-d6fcbc3d7ba3 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:26:03.618911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:03.619835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:26:03.620656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:26:03.620656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:26:03.620656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:03.623534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 13/28] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:03.630067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:03.661847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:03.795865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15c98e0-1651-4109-a3b1-d6fcbc3d7ba3 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:03.799303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:03.836863 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ec3f9-1929-4b0d-975f-3b513ddf1216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:03.837519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ec3f9-1929-4b0d-975f-3b513ddf1216 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:03.837796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ec3f9-1929-4b0d-975f-3b513ddf1216 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:03.838064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ec3f9-1929-4b0d-975f-3b513ddf1216 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:03.838940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ec3f9-1929-4b0d-975f-3b513ddf1216 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:03.840476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ec3f9-1929-4b0d-975f-3b513ddf1216 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:03.943931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a0452b-1a56-45f6-ba70-1ed08bc05dd8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:04.003297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 16/29] 10.222.0.77 () {60 vars in 1131 bytes} [Mon Aug 29 22:26:03 2022] POST /identity/v3/roles => generated 219 bytes in 394 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:04.005334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ec3f9-1929-4b0d-975f-3b513ddf1216 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:04.041057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491bbc60-7196-4f4b-a83d-e12175e6afa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:04.042260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491bbc60-7196-4f4b-a83d-e12175e6afa3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:04.042865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491bbc60-7196-4f4b-a83d-e12175e6afa3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:04.043347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491bbc60-7196-4f4b-a83d-e12175e6afa3 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:04.044826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491bbc60-7196-4f4b-a83d-e12175e6afa3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:04.045379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491bbc60-7196-4f4b-a83d-e12175e6afa3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:04.047228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491bbc60-7196-4f4b-a83d-e12175e6afa3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:04.073149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 14/30] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:26:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 443 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:04.085772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007617cd-67a0-4df2-951d-40e20df9379e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:04.087697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007617cd-67a0-4df2-951d-40e20df9379e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:04.089711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007617cd-67a0-4df2-951d-40e20df9379e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:04.093320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 15/31] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:04 2022] GET /identity => generated 270 bytes in 19 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:04.096545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007617cd-67a0-4df2-951d-40e20df9379e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:04.232684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ec3f9-1929-4b0d-975f-3b513ddf1216 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:04.304818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a4c3a6-72e9-4cae-a24a-f77daeeb8102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:04.305829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a4c3a6-72e9-4cae-a24a-f77daeeb8102 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:04.306269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a4c3a6-72e9-4cae-a24a-f77daeeb8102 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:04.306782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a4c3a6-72e9-4cae-a24a-f77daeeb8102 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:04.323996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a4c3a6-72e9-4cae-a24a-f77daeeb8102 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-35a4c3a6-72e9-4cae-a24a-f77daeeb8102 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:04.331158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:04.332074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:04.332869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:04.334368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:04.336445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 17/32] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:26:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:04.338910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276b1a73-7785-4e19-b379-fd9f258508ea None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:04.339414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276b1a73-7785-4e19-b379-fd9f258508ea None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:04.339878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276b1a73-7785-4e19-b379-fd9f258508ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:04.364859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-276b1a73-7785-4e19-b379-fd9f258508ea None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:04.367606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-276b1a73-7785-4e19-b379-fd9f258508ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:04.383579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:04.433455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007617cd-67a0-4df2-951d-40e20df9379e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:04.535913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 18/33] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:04.540290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d558ecf-0f81-4d6f-926b-f3ba5463cea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:04.542019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d558ecf-0f81-4d6f-926b-f3ba5463cea6 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:04.542251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276b1a73-7785-4e19-b379-fd9f258508ea None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:04.543517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d558ecf-0f81-4d6f-926b-f3ba5463cea6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:04.543995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d558ecf-0f81-4d6f-926b-f3ba5463cea6 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:04.545520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d558ecf-0f81-4d6f-926b-f3ba5463cea6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:04.548761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d558ecf-0f81-4d6f-926b-f3ba5463cea6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:04.797942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 16/34] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:26:04 2022] POST /identity/v3/projects => generated 301 bytes in 702 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:04.802947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2435e32e-227a-4dae-82aa-59a40b98571e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:04.803403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2435e32e-227a-4dae-82aa-59a40b98571e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:04.803907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2435e32e-227a-4dae-82aa-59a40b98571e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:04.806702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 17/35] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:04 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:04.825366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10991ed1-c0db-44b5-b9d2-001dd0197f5c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:04.825884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10991ed1-c0db-44b5-b9d2-001dd0197f5c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:04.826358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10991ed1-c0db-44b5-b9d2-001dd0197f5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:04.876898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-10991ed1-c0db-44b5-b9d2-001dd0197f5c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:04.881561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-10991ed1-c0db-44b5-b9d2-001dd0197f5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:04.903730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276b1a73-7785-4e19-b379-fd9f258508ea None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:04.950777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:04.952088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:04.952247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:04.952702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:04.953879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:04.954414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:04.957600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:04.979343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 19/36] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 441 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:04.980366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:04.989285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:05.029654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 18/37] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:05.031754 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f383f1e-7297-48c1-af70-0fb4cab95e16 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.032053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f383f1e-7297-48c1-af70-0fb4cab95e16 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.032329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f383f1e-7297-48c1-af70-0fb4cab95e16 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.033641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 19/38] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:05.039087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f383f1e-7297-48c1-af70-0fb4cab95e16 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:05.115437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:05.158456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ee37a4-8c31-440d-a769-e52ac00cf3bb None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:05.192620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f383f1e-7297-48c1-af70-0fb4cab95e16 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:05.221360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c00f84f-4855-4f8b-9796-dee50f98cd12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:05.221993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c00f84f-4855-4f8b-9796-dee50f98cd12 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.222277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c00f84f-4855-4f8b-9796-dee50f98cd12 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.222554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c00f84f-4855-4f8b-9796-dee50f98cd12 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.223285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c00f84f-4855-4f8b-9796-dee50f98cd12 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:05.225521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c00f84f-4855-4f8b-9796-dee50f98cd12 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:05.241181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 20/39] 10.222.0.77 () {60 vars in 1131 bytes} [Mon Aug 29 22:26:04 2022] POST /identity/v3/roles => generated 225 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:05.242011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ce81d2-a408-4544-bcab-b9cca3b9a8b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:05.242628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f5af7-e376-4d69-a07b-7a934aada0bf None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.243137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ce81d2-a408-4544-bcab-b9cca3b9a8b1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.243238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f5af7-e376-4d69-a07b-7a934aada0bf None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.243573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f5af7-e376-4d69-a07b-7a934aada0bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.243983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ce81d2-a408-4544-bcab-b9cca3b9a8b1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.244446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ce81d2-a408-4544-bcab-b9cca3b9a8b1 None admin] PATH_INFO: `/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.247775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ce81d2-a408-4544-bcab-b9cca3b9a8b1 None admin] RBAC: Authorizing `identity:get_user(user_id=2f1e799f7d5b4182a2e0ecd338aaa415)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:05.249649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ce81d2-a408-4544-bcab-b9cca3b9a8b1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:05.256552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 20/40] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:05 2022] GET /identity/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415 => generated 247 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:05.257854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ce81d2-a408-4544-bcab-b9cca3b9a8b1 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:05.275921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a14f5af7-e376-4d69-a07b-7a934aada0bf None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:05.277576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a14f5af7-e376-4d69-a07b-7a934aada0bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:05.371927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:05.385538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 21/41] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:05.387917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f5af7-e376-4d69-a07b-7a934aada0bf None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:05.531785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f5af7-e376-4d69-a07b-7a934aada0bf None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:05.541640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ce81d2-a408-4544-bcab-b9cca3b9a8b1 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:05.564432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:05.576902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71c496f-c357-4007-a396-e196725bd492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:05.577555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c496f-c357-4007-a396-e196725bd492 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.577837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c496f-c357-4007-a396-e196725bd492 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.578116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c496f-c357-4007-a396-e196725bd492 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.579305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71c496f-c357-4007-a396-e196725bd492 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:05.580770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71c496f-c357-4007-a396-e196725bd492 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:05.582584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 22/42] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:05.583648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb72b5f-d6c9-47a1-96ca-d82b1b995b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:05.584399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb72b5f-d6c9-47a1-96ca-d82b1b995b50 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.584686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb72b5f-d6c9-47a1-96ca-d82b1b995b50 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.584959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb72b5f-d6c9-47a1-96ca-d82b1b995b50 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.586879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb72b5f-d6c9-47a1-96ca-d82b1b995b50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:05.588436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb72b5f-d6c9-47a1-96ca-d82b1b995b50 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:05.590782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 21/43] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:05.590908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71c496f-c357-4007-a396-e196725bd492 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:05.595766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb72b5f-d6c9-47a1-96ca-d82b1b995b50 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:05.674670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:05.711461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb72b5f-d6c9-47a1-96ca-d82b1b995b50 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:05.714772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71c496f-c357-4007-a396-e196725bd492 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:05.750188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2508b174-492b-4533-8761-f182ac729f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:05.750838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508b174-492b-4533-8761-f182ac729f39 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.751219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508b174-492b-4533-8761-f182ac729f39 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.751495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508b174-492b-4533-8761-f182ac729f39 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.752176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2508b174-492b-4533-8761-f182ac729f39 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:05.752686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e0ff80-e7a3-4baf-b8b6-1c43ce16d6b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:05.753535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e0ff80-e7a3-4baf-b8b6-1c43ce16d6b2 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.753664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2508b174-492b-4533-8761-f182ac729f39 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:05.753820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e0ff80-e7a3-4baf-b8b6-1c43ce16d6b2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.754095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e0ff80-e7a3-4baf-b8b6-1c43ce16d6b2 None admin] PATH_INFO: `/v3/domains/default/users/2f1e799f7d5b4182a2e0ecd338aaa415/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.756088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e0ff80-e7a3-4baf-b8b6-1c43ce16d6b2 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=2f1e799f7d5b4182a2e0ecd338aaa415, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:05.757571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e0ff80-e7a3-4baf-b8b6-1c43ce16d6b2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:05.774184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 22/44] 10.222.0.77 () {58 vars in 1361 bytes} [Mon Aug 29 22:26:05 2022] PUT /identity/v3/domains/default/users/2f1e799f7d5b4182a2e0ecd338aaa415/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:05.775386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e0ff80-e7a3-4baf-b8b6-1c43ce16d6b2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:05.801349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 23/45] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:26:05 2022] POST /identity/v3/projects => generated 312 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:26:05.852300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:05.939465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e0ff80-e7a3-4baf-b8b6-1c43ce16d6b2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:05.985675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04ff771-0d6a-407e-851d-db364868eb30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:05.986318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04ff771-0d6a-407e-851d-db364868eb30 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:05.986606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04ff771-0d6a-407e-851d-db364868eb30 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:05.987135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04ff771-0d6a-407e-851d-db364868eb30 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:05.987672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04ff771-0d6a-407e-851d-db364868eb30 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:05.989158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04ff771-0d6a-407e-851d-db364868eb30 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:06.010136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 23/46] 10.222.0.77 () {60 vars in 1131 bytes} [Mon Aug 29 22:26:05 2022] POST /identity/v3/roles => generated 223 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:07.820434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542fc738-2ae8-46c3-aa33-063c5ab542c3 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:07.820844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542fc738-2ae8-46c3-aa33-063c5ab542c3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:07.821010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542fc738-2ae8-46c3-aa33-063c5ab542c3 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:07.822849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 24/47] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:07.828484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ea5c32-e452-47d6-ad8b-df1f40341073 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:07.828937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ea5c32-e452-47d6-ad8b-df1f40341073 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:07.829045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ea5c32-e452-47d6-ad8b-df1f40341073 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:07.848364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-55ea5c32-e452-47d6-ad8b-df1f40341073 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:07.849584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-55ea5c32-e452-47d6-ad8b-df1f40341073 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:07.864871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a5f54-1e8f-4223-81c1-e22f0b07e2b7 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:07.865157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a5f54-1e8f-4223-81c1-e22f0b07e2b7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:07.865433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a5f54-1e8f-4223-81c1-e22f0b07e2b7 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:07.866816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 25/48] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:07.872385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd48e59f-ad6e-4d5a-9b5c-dbcc3db471ed None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:07.872666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd48e59f-ad6e-4d5a-9b5c-dbcc3db471ed None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:07.872942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd48e59f-ad6e-4d5a-9b5c-dbcc3db471ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:07.893352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-cd48e59f-ad6e-4d5a-9b5c-dbcc3db471ed None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:07.893644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cd48e59f-ad6e-4d5a-9b5c-dbcc3db471ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:07.933448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 24/49] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:07.965589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 26/50] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:08.203757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ea5c32-e452-47d6-ad8b-df1f40341073 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:08.246649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd48e59f-ad6e-4d5a-9b5c-dbcc3db471ed None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:08.248845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:08.297590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:08.319616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ea5c32-e452-47d6-ad8b-df1f40341073 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:08.359023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20370f4a-0601-43a4-b818-e2bc38505581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:08.359675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20370f4a-0601-43a4-b818-e2bc38505581 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:08.359962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20370f4a-0601-43a4-b818-e2bc38505581 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:08.360242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20370f4a-0601-43a4-b818-e2bc38505581 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:08.364779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd48e59f-ad6e-4d5a-9b5c-dbcc3db471ed None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:08.372477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20370f4a-0601-43a4-b818-e2bc38505581 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-20370f4a-0601-43a4-b818-e2bc38505581 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:08.379810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:08.380908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:08.381926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 25/51] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:26:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:08.387326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20370f4a-0601-43a4-b818-e2bc38505581 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:08.410886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1100583-959d-44bd-a02f-f036b7f3d309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:08.411565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1100583-959d-44bd-a02f-f036b7f3d309 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:08.411847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1100583-959d-44bd-a02f-f036b7f3d309 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:08.412114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1100583-959d-44bd-a02f-f036b7f3d309 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:08.413262 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1100583-959d-44bd-a02f-f036b7f3d309 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:08.415022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1100583-959d-44bd-a02f-f036b7f3d309 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:08.417146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 27/52] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:08.422009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1100583-959d-44bd-a02f-f036b7f3d309 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:08.498160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:08.501660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20370f4a-0601-43a4-b818-e2bc38505581 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:08.535695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1100583-959d-44bd-a02f-f036b7f3d309 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:08.538428 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:08.539190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:08.539408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:08.539690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:08.540384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:08.540692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:08.542124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:08.557441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:08.557644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 26/53] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:08.563233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:08.582034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0aee7a-0b1f-4756-bbde-14fcf0bea415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:08.582656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0aee7a-0b1f-4756-bbde-14fcf0bea415 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:08.582980 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0aee7a-0b1f-4756-bbde-14fcf0bea415 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:08.583249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0aee7a-0b1f-4756-bbde-14fcf0bea415 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:08.583930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0aee7a-0b1f-4756-bbde-14fcf0bea415 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:08.585408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0aee7a-0b1f-4756-bbde-14fcf0bea415 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:08.616488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 28/54] 10.222.0.77 () {60 vars in 1140 bytes} [Mon Aug 29 22:26:08 2022] POST /identity/v3/projects => generated 298 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:26:08.689496 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ba14b7-96d9-4e2f-b5c1-d3abff45e2fa None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:08.729301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90a7b91-6457-475f-b6eb-52a9e7a3911e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:08.729980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a7b91-6457-475f-b6eb-52a9e7a3911e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:08.730266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a7b91-6457-475f-b6eb-52a9e7a3911e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:08.730547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a7b91-6457-475f-b6eb-52a9e7a3911e None admin] PATH_INFO: `/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:08.731836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90a7b91-6457-475f-b6eb-52a9e7a3911e None admin] RBAC: Authorizing `identity:get_user(user_id=2f1e799f7d5b4182a2e0ecd338aaa415)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:08.733361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90a7b91-6457-475f-b6eb-52a9e7a3911e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:08.735190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 27/55] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:08 2022] GET /identity/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415 => generated 247 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:08.740984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0aee7a-0b1f-4756-bbde-14fcf0bea415 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:08.874378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0aee7a-0b1f-4756-bbde-14fcf0bea415 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:08.923695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4976e2-a678-4966-aae9-3e496ebb499b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:08.924539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4976e2-a678-4966-aae9-3e496ebb499b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:08.924890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4976e2-a678-4966-aae9-3e496ebb499b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:08.925237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4976e2-a678-4966-aae9-3e496ebb499b None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:08.927321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4976e2-a678-4966-aae9-3e496ebb499b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:08.928824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4976e2-a678-4966-aae9-3e496ebb499b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:08.931050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 29/56] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:08.937050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90a7b91-6457-475f-b6eb-52a9e7a3911e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:08.967798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:09.059018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90a7b91-6457-475f-b6eb-52a9e7a3911e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:09.105182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4466ca85-8ae6-45b4-b224-67efeb059fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:09.105722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4466ca85-8ae6-45b4-b224-67efeb059fb3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:09.106000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4466ca85-8ae6-45b4-b224-67efeb059fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:09.106277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4466ca85-8ae6-45b4-b224-67efeb059fb3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:09.107049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4466ca85-8ae6-45b4-b224-67efeb059fb3 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.domain.id=default, user.id=2f1e799f7d5b4182a2e0ecd338aaa415) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:09.107357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4466ca85-8ae6-45b4-b224-67efeb059fb3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:09.108829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4466ca85-8ae6-45b4-b224-67efeb059fb3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:09.115653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 28/57] 10.222.0.77 () {56 vars in 1322 bytes} [Mon Aug 29 22:26:08 2022] GET /identity/v3/role_assignments?user.id=2f1e799f7d5b4182a2e0ecd338aaa415&scope.domain.id=default&role.id=38171c89968943bd92d9e9758db65615 => generated 530 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:10.594718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d32bd7-416f-40b7-847e-0b73c44e28d4 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:10.595333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d32bd7-416f-40b7-847e-0b73c44e28d4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:10.595333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d32bd7-416f-40b7-847e-0b73c44e28d4 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:10.596652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 30/58] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:10.602940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e89a30-2aa6-4f0f-9b91-5753b13d6972 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:10.603441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e89a30-2aa6-4f0f-9b91-5753b13d6972 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:10.603551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e89a30-2aa6-4f0f-9b91-5753b13d6972 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:10.627352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c9e89a30-2aa6-4f0f-9b91-5753b13d6972 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:10.628597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c9e89a30-2aa6-4f0f-9b91-5753b13d6972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:10.706096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 29/59] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:11.002546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d32bd7-416f-40b7-847e-0b73c44e28d4 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:11.036359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:11.114679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d32bd7-416f-40b7-847e-0b73c44e28d4 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:11.152283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0a733a-7939-4f12-b6fa-049193be0e37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:11.152929 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0a733a-7939-4f12-b6fa-049193be0e37 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:11.153216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0a733a-7939-4f12-b6fa-049193be0e37 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:11.153495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0a733a-7939-4f12-b6fa-049193be0e37 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:11.154664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0a733a-7939-4f12-b6fa-049193be0e37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:11.156187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0a733a-7939-4f12-b6fa-049193be0e37 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:11.157799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 31/60] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:11.163478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e89a30-2aa6-4f0f-9b91-5753b13d6972 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:11.189177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:11.297586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e89a30-2aa6-4f0f-9b91-5753b13d6972 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:11.351701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c9eaaa-24e9-4ce4-9839-846c88865df7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:11.352456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9eaaa-24e9-4ce4-9839-846c88865df7 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:11.352857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9eaaa-24e9-4ce4-9839-846c88865df7 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:11.353133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9eaaa-24e9-4ce4-9839-846c88865df7 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:11.353818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c9eaaa-24e9-4ce4-9839-846c88865df7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:11.355360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c9eaaa-24e9-4ce4-9839-846c88865df7 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:11.395020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 30/61] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:26:11 2022] POST /identity/v3/users => generated 275 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:14.980285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd4ebe3-0abe-46ae-bb25-a19eb5536de8 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:14.980843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd4ebe3-0abe-46ae-bb25-a19eb5536de8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:14.981147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd4ebe3-0abe-46ae-bb25-a19eb5536de8 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:14.983360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 32/62] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:14.990142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3493c1-f32b-4f21-a536-0cac2b2f31e5 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:14.990708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3493c1-f32b-4f21-a536-0cac2b2f31e5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:14.991065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3493c1-f32b-4f21-a536-0cac2b2f31e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:15.004451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29312903-b377-4309-adf6-fde46794c944 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:15.004451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29312903-b377-4309-adf6-fde46794c944 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:15.004451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29312903-b377-4309-adf6-fde46794c944 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:15.004451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 33/63] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:15.013889 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f46cf9c-86c1-410b-8309-cfe86c6bda97 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:15.014711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f46cf9c-86c1-410b-8309-cfe86c6bda97 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:15.015498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f46cf9c-86c1-410b-8309-cfe86c6bda97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:15.022493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9a3493c1-f32b-4f21-a536-0cac2b2f31e5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:15.026284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9a3493c1-f32b-4f21-a536-0cac2b2f31e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:15.062868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2f46cf9c-86c1-410b-8309-cfe86c6bda97 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:15.065489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2f46cf9c-86c1-410b-8309-cfe86c6bda97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:15.251485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 31/64] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:15.298916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 34/65] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:15.332303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b5a64e-4bdb-4a70-9d88-1c204e905f4c None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:15.333154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b5a64e-4bdb-4a70-9d88-1c204e905f4c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:15.333881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b5a64e-4bdb-4a70-9d88-1c204e905f4c None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:15.339307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 32/66] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:15 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:15.348391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947fb8b3-e0d8-443a-9bbc-490f18dcc68b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:15.349238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947fb8b3-e0d8-443a-9bbc-490f18dcc68b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:15.349956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947fb8b3-e0d8-443a-9bbc-490f18dcc68b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:15.394569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-947fb8b3-e0d8-443a-9bbc-490f18dcc68b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:15.396988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-947fb8b3-e0d8-443a-9bbc-490f18dcc68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:15.448293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:15.635919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b5a64e-4bdb-4a70-9d88-1c204e905f4c None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:15.662561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:15.741671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 35/67] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:15.756421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09638c42-cd41-4f30-ae23-05dd50f4b69c None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:15.757500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09638c42-cd41-4f30-ae23-05dd50f4b69c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:15.759157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09638c42-cd41-4f30-ae23-05dd50f4b69c None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:15.762762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 36/68] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:15 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:15.774800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0845460-eda0-4678-8726-eaa1c26aaa54 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:15.774800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0845460-eda0-4678-8726-eaa1c26aaa54 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:15.775211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0845460-eda0-4678-8726-eaa1c26aaa54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:15.833176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d0845460-eda0-4678-8726-eaa1c26aaa54 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:15.835316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d0845460-eda0-4678-8726-eaa1c26aaa54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:15.963432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b5a64e-4bdb-4a70-9d88-1c204e905f4c None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:15.982381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 37/69] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:15.987942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0845460-eda0-4678-8726-eaa1c26aaa54 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.027913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb52e25-3eef-4797-b970-ae7ba0358194 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.028574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb52e25-3eef-4797-b970-ae7ba0358194 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.028956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb52e25-3eef-4797-b970-ae7ba0358194 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.029277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb52e25-3eef-4797-b970-ae7ba0358194 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.045950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb52e25-3eef-4797-b970-ae7ba0358194 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-ebb52e25-3eef-4797-b970-ae7ba0358194 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:16.051626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:16.053173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:16.054066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 33/70] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:26:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 418 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:26:16.055453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34425306-3093-4b60-8686-bcadee60502f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.055957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34425306-3093-4b60-8686-bcadee60502f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.056402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34425306-3093-4b60-8686-bcadee60502f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.058948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 34/71] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:16.067521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34425306-3093-4b60-8686-bcadee60502f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.136147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:16.149465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0845460-eda0-4678-8726-eaa1c26aaa54 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:16.181045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:16.205535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae54f929-ae89-417e-82a4-ac8ac7b96d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.206186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae54f929-ae89-417e-82a4-ac8ac7b96d68 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.206473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae54f929-ae89-417e-82a4-ac8ac7b96d68 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.206777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae54f929-ae89-417e-82a4-ac8ac7b96d68 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.220235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae54f929-ae89-417e-82a4-ac8ac7b96d68 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:16.224609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34425306-3093-4b60-8686-bcadee60502f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-ae54f929-ae89-417e-82a4-ac8ac7b96d68 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:16.225172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:16.226082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 38/72] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:16.233937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9834cbab-2369-4dac-ba72-e464bd25fe42 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.234609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9834cbab-2369-4dac-ba72-e464bd25fe42 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.234609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9834cbab-2369-4dac-ba72-e464bd25fe42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.252427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-9834cbab-2369-4dac-ba72-e464bd25fe42 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:16.253748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9834cbab-2369-4dac-ba72-e464bd25fe42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:16.281063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.282215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.282943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.283347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.284382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:16.285126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:16.287131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:16.300880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 35/73] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:16.301734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.336856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 39/74] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:16.337724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9834cbab-2369-4dac-ba72-e464bd25fe42 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.434265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7e4ea0-c66c-4ed4-8549-3cafb5a6f0bc None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:16.438207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:16.460531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9834cbab-2369-4dac-ba72-e464bd25fe42 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:16.481810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:16.491026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cbc32f-0942-4517-aa22-8cffd88c3d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.491871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cbc32f-0942-4517-aa22-8cffd88c3d54 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.493045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cbc32f-0942-4517-aa22-8cffd88c3d54 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.493674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cbc32f-0942-4517-aa22-8cffd88c3d54 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.504680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cbc32f-0942-4517-aa22-8cffd88c3d54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:16.505680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.506396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.506751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.507043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.507850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:16.508235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:16.509890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-72cbc32f-0942-4517-aa22-8cffd88c3d54 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:16.511802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:16.512748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:16.513668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:16.515616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 36/75] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:16.516549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cbc32f-0942-4517-aa22-8cffd88c3d54 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.525741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 40/76] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:16.527738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.638338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cbc32f-0942-4517-aa22-8cffd88c3d54 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:16.641530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1135dd-ec80-4095-8f5f-2c12217ffdb8 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:16.677046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2b6764-bda9-4fb5-9815-008731723267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.677757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b6764-bda9-4fb5-9815-008731723267 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.677870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a6f5cf-c34a-426c-8e8d-2101215b0e24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.678060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b6764-bda9-4fb5-9815-008731723267 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.678393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a6f5cf-c34a-426c-8e8d-2101215b0e24 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.678499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b6764-bda9-4fb5-9815-008731723267 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.678677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a6f5cf-c34a-426c-8e8d-2101215b0e24 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.678983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a6f5cf-c34a-426c-8e8d-2101215b0e24 None admin] PATH_INFO: `/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.680693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a6f5cf-c34a-426c-8e8d-2101215b0e24 None admin] RBAC: Authorizing `identity:get_user(user_id=2f1e799f7d5b4182a2e0ecd338aaa415)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:16.682189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a6f5cf-c34a-426c-8e8d-2101215b0e24 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:16.683850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 41/77] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415 => generated 247 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:16.684731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a6f5cf-c34a-426c-8e8d-2101215b0e24 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.692236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2b6764-bda9-4fb5-9815-008731723267 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-7b2b6764-bda9-4fb5-9815-008731723267 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:16.698086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:16.699210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:16.700719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 37/78] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:16.701796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:16.701977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2b6764-bda9-4fb5-9815-008731723267 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.754840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:16.814041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a6f5cf-c34a-426c-8e8d-2101215b0e24 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:16.816966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2b6764-bda9-4fb5-9815-008731723267 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:16.853294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.853910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.854197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.854471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.855197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:16.855501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:16.856922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:16.863290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a662306-94a8-4539-8410-096f08716ba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:16.863933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a662306-94a8-4539-8410-096f08716ba5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:16.864224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a662306-94a8-4539-8410-096f08716ba5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:16.864502 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a662306-94a8-4539-8410-096f08716ba5 None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:16.866385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 42/79] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:16.867384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.883095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a662306-94a8-4539-8410-096f08716ba5 None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:16.884605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a662306-94a8-4539-8410-096f08716ba5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:16.886466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 38/80] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:16.887188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a662306-94a8-4539-8410-096f08716ba5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:16.949760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:17.000117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbe48bf-1453-4e90-8aab-f74848f5faa9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:17.045989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab463c5e-13f2-4488-8847-d35b1bd924b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:17.046630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab463c5e-13f2-4488-8847-d35b1bd924b2 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:17.046895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab463c5e-13f2-4488-8847-d35b1bd924b2 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:17.047925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab463c5e-13f2-4488-8847-d35b1bd924b2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:17.049371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab463c5e-13f2-4488-8847-d35b1bd924b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:17.051308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab463c5e-13f2-4488-8847-d35b1bd924b2 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:17.054040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 43/81] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:17.056722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab463c5e-13f2-4488-8847-d35b1bd924b2 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:17.066207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a662306-94a8-4539-8410-096f08716ba5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:17.080705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:17.115722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341465db-e674-46df-abd9-14488d872c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:17.116560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341465db-e674-46df-abd9-14488d872c88 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:17.116847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341465db-e674-46df-abd9-14488d872c88 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:17.117256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341465db-e674-46df-abd9-14488d872c88 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:17.213874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341465db-e674-46df-abd9-14488d872c88 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:17.215430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341465db-e674-46df-abd9-14488d872c88 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:17.233346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab463c5e-13f2-4488-8847-d35b1bd924b2 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:17.272252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:17.273120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:17.273120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:17.274469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:17.274941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:17.275236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:17.277096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:17.294748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 44/82] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:17.294748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:17.294748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:17.364379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 39/83] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:17.365170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341465db-e674-46df-abd9-14488d872c88 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:17.381510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:17.434598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d8da57-78ba-43fa-ad41-799b0ed3231f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:17.477655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87572b07-7985-444a-ac4f-c13147160167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:17.478309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87572b07-7985-444a-ac4f-c13147160167 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:17.478597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87572b07-7985-444a-ac4f-c13147160167 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:17.478979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87572b07-7985-444a-ac4f-c13147160167 None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:17.480204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87572b07-7985-444a-ac4f-c13147160167 None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:17.482779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87572b07-7985-444a-ac4f-c13147160167 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:17.483371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 45/84] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:17 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:17.484154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87572b07-7985-444a-ac4f-c13147160167 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:17.506380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341465db-e674-46df-abd9-14488d872c88 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:17.550334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b474a1-792f-4a51-ad38-297a17494ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:17.551367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b474a1-792f-4a51-ad38-297a17494ee2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:17.551831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b474a1-792f-4a51-ad38-297a17494ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:17.552312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b474a1-792f-4a51-ad38-297a17494ee2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:17.563629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:17.601842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87572b07-7985-444a-ac4f-c13147160167 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:17.643168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d520642-1168-4657-87a9-46ca3a7c2e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:17.643712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d520642-1168-4657-87a9-46ca3a7c2e70 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:17.643997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d520642-1168-4657-87a9-46ca3a7c2e70 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:17.644274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d520642-1168-4657-87a9-46ca3a7c2e70 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:17.644953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d520642-1168-4657-87a9-46ca3a7c2e70 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:17.645863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b474a1-792f-4a51-ad38-297a17494ee2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:17.646432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d520642-1168-4657-87a9-46ca3a7c2e70 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:17.647384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b474a1-792f-4a51-ad38-297a17494ee2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:17.658858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:17.676331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 46/85] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:26:17 2022] POST /identity/v3/users => generated 289 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:17.677202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d520642-1168-4657-87a9-46ca3a7c2e70 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:17.800662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 40/86] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:17.807064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b474a1-792f-4a51-ad38-297a17494ee2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:17.829137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d520642-1168-4657-87a9-46ca3a7c2e70 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:17.880683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:17.900396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f272e7-e16d-42bf-8573-61b090fff25f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:17.901245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f272e7-e16d-42bf-8573-61b090fff25f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:17.901527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f272e7-e16d-42bf-8573-61b090fff25f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:17.901957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f272e7-e16d-42bf-8573-61b090fff25f None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:17.903488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f272e7-e16d-42bf-8573-61b090fff25f None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:17.904686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f272e7-e16d-42bf-8573-61b090fff25f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:17.906389 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 47/87] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:17 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:17.907251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f272e7-e16d-42bf-8573-61b090fff25f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:17.957713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b474a1-792f-4a51-ad38-297a17494ee2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:17.975008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:18.021547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d4d0a8-2834-4cb5-a438-2766cadcc0d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:18.022677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d4d0a8-2834-4cb5-a438-2766cadcc0d1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:18.022677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d4d0a8-2834-4cb5-a438-2766cadcc0d1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:18.022865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d4d0a8-2834-4cb5-a438-2766cadcc0d1 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:18.024093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d4d0a8-2834-4cb5-a438-2766cadcc0d1 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:18.025852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d4d0a8-2834-4cb5-a438-2766cadcc0d1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:18.027617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 41/88] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:17 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:18.028443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d4d0a8-2834-4cb5-a438-2766cadcc0d1 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:18.076093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f272e7-e16d-42bf-8573-61b090fff25f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:18.126322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd175dd2-1f7d-4720-95f2-f3a24a434e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:18.127154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd175dd2-1f7d-4720-95f2-f3a24a434e51 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:18.127447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd175dd2-1f7d-4720-95f2-f3a24a434e51 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:18.127726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd175dd2-1f7d-4720-95f2-f3a24a434e51 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:18.222161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d4d0a8-2834-4cb5-a438-2766cadcc0d1 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:18.234003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:18.245955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd175dd2-1f7d-4720-95f2-f3a24a434e51 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:18.247913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd175dd2-1f7d-4720-95f2-f3a24a434e51 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:18.313922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c95d74-d5e1-4dda-a518-9477cbce3c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:18.315620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c95d74-d5e1-4dda-a518-9477cbce3c94 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:18.316106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c95d74-d5e1-4dda-a518-9477cbce3c94 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:18.316571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c95d74-d5e1-4dda-a518-9477cbce3c94 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:18.318978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c95d74-d5e1-4dda-a518-9477cbce3c94 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:18.322194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c95d74-d5e1-4dda-a518-9477cbce3c94 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:18.327103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 42/89] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:18 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:18.329015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c95d74-d5e1-4dda-a518-9477cbce3c94 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:18.408710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:18.422449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 48/90] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 509 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:18.422449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd175dd2-1f7d-4720-95f2-f3a24a434e51 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:18.496426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c95d74-d5e1-4dda-a518-9477cbce3c94 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:18.557115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4688ca-4b7d-4a98-a1e3-a2a47876c0e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:18.558153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4688ca-4b7d-4a98-a1e3-a2a47876c0e2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:18.558635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4688ca-4b7d-4a98-a1e3-a2a47876c0e2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:18.559336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4688ca-4b7d-4a98-a1e3-a2a47876c0e2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:18.562917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:18.579007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd175dd2-1f7d-4720-95f2-f3a24a434e51 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:18.657332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:18.658881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:18.659377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:18.659861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:18.661915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=bad66f048c724ae380da0250ba68d56d, user.id=2f1e799f7d5b4182a2e0ecd338aaa415) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:18.662477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:18.666682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:18.672366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4688ca-4b7d-4a98-a1e3-a2a47876c0e2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:18.675262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4688ca-4b7d-4a98-a1e3-a2a47876c0e2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:18.686897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 49/91] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:18 2022] GET /identity/v3/role_assignments?user.id=2f1e799f7d5b4182a2e0ecd338aaa415&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=38171c89968943bd92d9e9758db65615 => generated 259 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:18.688258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:18.777546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:18.885507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 43/92] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 558 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:18.886933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4688ca-4b7d-4a98-a1e3-a2a47876c0e2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:18.932393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c950659-8ee1-45dd-8d43-241b239e40cd None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:18.976480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:18.977147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:18.977475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:18.977754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:18.978504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] RBAC: Adding query filter params (role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=bad66f048c724ae380da0250ba68d56d, user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:18.979436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:18.981432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:18.991621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 50/93] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:18 2022] GET /identity/v3/role_assignments?user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=0eb5172435534daf81351f9eb5281719 => generated 259 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:18.992892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:19.049814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:19.137759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4688ca-4b7d-4a98-a1e3-a2a47876c0e2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:19.167146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66075c29-d5ac-41ef-af41-34566a15adec None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:19.224965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b6490e-30bf-4aa8-bb4f-6d6168c91e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:19.225642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b6490e-30bf-4aa8-bb4f-6d6168c91e41 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:19.225932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b6490e-30bf-4aa8-bb4f-6d6168c91e41 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:19.226205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b6490e-30bf-4aa8-bb4f-6d6168c91e41 None admin] PATH_INFO: `/v3/projects/adcafe4c8eaf49bda38a2fe27ad5bbf4` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:19.229353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b6490e-30bf-4aa8-bb4f-6d6168c91e41 None admin] RBAC: Authorizing `identity:get_project(project_id=adcafe4c8eaf49bda38a2fe27ad5bbf4)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:19.230953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b6490e-30bf-4aa8-bb4f-6d6168c91e41 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:19.232992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 51/94] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:18 2022] GET /identity/v3/projects/adcafe4c8eaf49bda38a2fe27ad5bbf4 => generated 312 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:19.239610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b6490e-30bf-4aa8-bb4f-6d6168c91e41 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:19.243913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef19bd5-f49c-494d-8b13-d9e2736120cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:19.244585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef19bd5-f49c-494d-8b13-d9e2736120cf None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:19.244871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef19bd5-f49c-494d-8b13-d9e2736120cf None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:19.245148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef19bd5-f49c-494d-8b13-d9e2736120cf None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:19.254959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef19bd5-f49c-494d-8b13-d9e2736120cf None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:19.256790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef19bd5-f49c-494d-8b13-d9e2736120cf None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:19.258783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 44/95] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:18 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 373 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:19.265452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef19bd5-f49c-494d-8b13-d9e2736120cf None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:19.299254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:19.415796 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef19bd5-f49c-494d-8b13-d9e2736120cf None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:19.465053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b54f43-4975-4263-b4b7-e6c92776d3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:19.465682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b54f43-4975-4263-b4b7-e6c92776d3ae None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:19.465965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b54f43-4975-4263-b4b7-e6c92776d3ae None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:19.466239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b54f43-4975-4263-b4b7-e6c92776d3ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:19.467018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b54f43-4975-4263-b4b7-e6c92776d3ae None admin] RBAC: Adding query filter params (role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=bad66f048c724ae380da0250ba68d56d, user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:19.467341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b54f43-4975-4263-b4b7-e6c92776d3ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:19.468803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b54f43-4975-4263-b4b7-e6c92776d3ae None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:19.475321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 45/96] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:19 2022] GET /identity/v3/role_assignments?user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 259 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:19.534167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:19.553984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b6490e-30bf-4aa8-bb4f-6d6168c91e41 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:19.621339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b862fe66-365d-492c-853f-dd1d5fccce7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:19.621971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862fe66-365d-492c-853f-dd1d5fccce7a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:19.622253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862fe66-365d-492c-853f-dd1d5fccce7a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:19.622528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862fe66-365d-492c-853f-dd1d5fccce7a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:19.623301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b862fe66-365d-492c-853f-dd1d5fccce7a None admin] RBAC: Adding query filter params (role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=adcafe4c8eaf49bda38a2fe27ad5bbf4, user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:19.623612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b862fe66-365d-492c-853f-dd1d5fccce7a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:19.625062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b862fe66-365d-492c-853f-dd1d5fccce7a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:19.636133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 52/97] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:19 2022] GET /identity/v3/role_assignments?user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1&scope.project.id=adcafe4c8eaf49bda38a2fe27ad5bbf4&role.id=0eb5172435534daf81351f9eb5281719 => generated 259 bytes in 397 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:20.241471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968b52fc-a938-4c35-bb14-77a24dca9245 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:20.241766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968b52fc-a938-4c35-bb14-77a24dca9245 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:20.242025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968b52fc-a938-4c35-bb14-77a24dca9245 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:20.245060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 46/98] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:20.251021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241bbdea-0250-4a75-a4a4-46294fff311d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:20.251758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241bbdea-0250-4a75-a4a4-46294fff311d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:20.251758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241bbdea-0250-4a75-a4a4-46294fff311d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:20.277497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-241bbdea-0250-4a75-a4a4-46294fff311d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:20.278787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-241bbdea-0250-4a75-a4a4-46294fff311d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:20.398313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 53/99] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:20.820350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be9b8f0-d5a6-4b50-9564-818abfbfe017 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:20.823638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be9b8f0-d5a6-4b50-9564-818abfbfe017 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:20.823638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be9b8f0-d5a6-4b50-9564-818abfbfe017 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:20.823638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 47/100] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:20.840856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418f1c92-79e0-4256-9be5-494cf0776811 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:20.841128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418f1c92-79e0-4256-9be5-494cf0776811 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:20.841418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418f1c92-79e0-4256-9be5-494cf0776811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:20.867450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-418f1c92-79e0-4256-9be5-494cf0776811 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:20.869024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-418f1c92-79e0-4256-9be5-494cf0776811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:20.984569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 54/101] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:21.103669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be9b8f0-d5a6-4b50-9564-818abfbfe017 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:21.217072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:21.251726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be9b8f0-d5a6-4b50-9564-818abfbfe017 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:21.304292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d791e9f3-597f-44a7-b54f-00a074873f8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:21.305836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d791e9f3-597f-44a7-b54f-00a074873f8a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:21.306285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d791e9f3-597f-44a7-b54f-00a074873f8a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:21.306750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d791e9f3-597f-44a7-b54f-00a074873f8a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:21.331832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d791e9f3-597f-44a7-b54f-00a074873f8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-d791e9f3-597f-44a7-b54f-00a074873f8a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:21.344083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:21.346243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:21.347924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 48/102] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:21.356599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418f1c92-79e0-4256-9be5-494cf0776811 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:21.400618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:21.566067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418f1c92-79e0-4256-9be5-494cf0776811 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:21.619331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:21.620325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:21.620760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:21.621045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:21.621752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] RBAC: Adding query filter params (name=reader) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:21.622064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:21.623514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:21.633675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 55/103] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:21.640044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d791e9f3-597f-44a7-b54f-00a074873f8a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:21.652172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:21.803817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:21.886917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feb07bd-8c2d-432e-aba7-fe72eb184add None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:21.911664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d791e9f3-597f-44a7-b54f-00a074873f8a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:21.950793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03b7a60-250b-4bff-bcc7-b77d18624c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:21.951511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03b7a60-250b-4bff-bcc7-b77d18624c68 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:21.951850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03b7a60-250b-4bff-bcc7-b77d18624c68 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:21.952126 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03b7a60-250b-4bff-bcc7-b77d18624c68 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:21.953782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03b7a60-250b-4bff-bcc7-b77d18624c68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:21.955485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03b7a60-250b-4bff-bcc7-b77d18624c68 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:21.957540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 56/104] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:21.965443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03b7a60-250b-4bff-bcc7-b77d18624c68 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:21.980449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7034ca81-6b9b-4680-8300-18a6f01783ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:21.981109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7034ca81-6b9b-4680-8300-18a6f01783ee None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:21.981388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7034ca81-6b9b-4680-8300-18a6f01783ee None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:21.981734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7034ca81-6b9b-4680-8300-18a6f01783ee None admin] PATH_INFO: `/v3/users/e65a84069e1d4cd7b0759a34dbfbe8dd` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:21.985865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:21.999757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7034ca81-6b9b-4680-8300-18a6f01783ee None admin] RBAC: Authorizing `identity:get_user(user_id=e65a84069e1d4cd7b0759a34dbfbe8dd)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:22.001245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7034ca81-6b9b-4680-8300-18a6f01783ee None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:22.003284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 49/105] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:21 2022] GET /identity/v3/users/e65a84069e1d4cd7b0759a34dbfbe8dd => generated 289 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:22.004650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c430ee6d-a527-46fe-984d-b93f03ae82f9 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.004939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c430ee6d-a527-46fe-984d-b93f03ae82f9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.005219 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c430ee6d-a527-46fe-984d-b93f03ae82f9 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.006770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 50/106] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:22.011879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c430ee6d-a527-46fe-984d-b93f03ae82f9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:22.166279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03b7a60-250b-4bff-bcc7-b77d18624c68 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:22.223874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c430ee6d-a527-46fe-984d-b93f03ae82f9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:22.226677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5a7222-8024-4619-9105-6b7052b64e23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:22.227948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a7222-8024-4619-9105-6b7052b64e23 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.228231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a7222-8024-4619-9105-6b7052b64e23 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.228508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a7222-8024-4619-9105-6b7052b64e23 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.229192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5a7222-8024-4619-9105-6b7052b64e23 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:22.230693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5a7222-8024-4619-9105-6b7052b64e23 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:22.269775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:22.289169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 57/107] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:26:21 2022] POST /identity/v3/projects => generated 302 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:26:22.291352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea59bf0-04a4-4ed9-a749-2acd10847283 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.291645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea59bf0-04a4-4ed9-a749-2acd10847283 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.291937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea59bf0-04a4-4ed9-a749-2acd10847283 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.300578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2640d1f-e2cf-469d-8d22-e313d5bb34df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:22.301237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2640d1f-e2cf-469d-8d22-e313d5bb34df None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.301626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2640d1f-e2cf-469d-8d22-e313d5bb34df None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.301933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2640d1f-e2cf-469d-8d22-e313d5bb34df None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.331293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3ea59bf0-04a4-4ed9-a749-2acd10847283 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:22.334373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3ea59bf0-04a4-4ed9-a749-2acd10847283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:22.453367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:22.471882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2640d1f-e2cf-469d-8d22-e313d5bb34df None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:22.473697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2640d1f-e2cf-469d-8d22-e313d5bb34df None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:22.524871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 58/108] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:22.529552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836ee1d7-fe88-42c8-bc23-ebb7396de723 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.530428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836ee1d7-fe88-42c8-bc23-ebb7396de723 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.532461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836ee1d7-fe88-42c8-bc23-ebb7396de723 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.534711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 59/109] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:22 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:22.545515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb446f-d3dd-473d-89a4-284505e2d4e9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.546118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb446f-d3dd-473d-89a4-284505e2d4e9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.546715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb446f-d3dd-473d-89a4-284505e2d4e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.589169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1ddb446f-d3dd-473d-89a4-284505e2d4e9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:22.591555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1ddb446f-d3dd-473d-89a4-284505e2d4e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:22.704090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 60/110] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:22.733736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 51/111] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 722 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:22.739425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddb446f-d3dd-473d-89a4-284505e2d4e9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:22.824392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:22.889229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddb446f-d3dd-473d-89a4-284505e2d4e9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:22.931945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb75a367-b976-4c41-9cd2-696b6ea7b37e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:22.932693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75a367-b976-4c41-9cd2-696b6ea7b37e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.933052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75a367-b976-4c41-9cd2-696b6ea7b37e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.933435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75a367-b976-4c41-9cd2-696b6ea7b37e None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.935343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb75a367-b976-4c41-9cd2-696b6ea7b37e None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:22.937957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb75a367-b976-4c41-9cd2-696b6ea7b37e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:22.940504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 61/112] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:22 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:22.947759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2640d1f-e2cf-469d-8d22-e313d5bb34df None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:22.952157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239f684-5e54-4ed8-8317-4e81244be1d1 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.952441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239f684-5e54-4ed8-8317-4e81244be1d1 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.952783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239f684-5e54-4ed8-8317-4e81244be1d1 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.954464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 62/113] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:22.964416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e12cde-9451-4d83-bfc1-07ffa2a60d86 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:22.964714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e12cde-9451-4d83-bfc1-07ffa2a60d86 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:22.964969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e12cde-9451-4d83-bfc1-07ffa2a60d86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:22.999449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-63e12cde-9451-4d83-bfc1-07ffa2a60d86 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:23.000992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-63e12cde-9451-4d83-bfc1-07ffa2a60d86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:23.104058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 63/114] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:23.105576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e12cde-9451-4d83-bfc1-07ffa2a60d86 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:23.159284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:23.172744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2640d1f-e2cf-469d-8d22-e313d5bb34df None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:23.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:23.254681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:23.255524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:23.255815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:23.256102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:23.256865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] RBAC: Adding query filter params (role.id=bcd685ff205446cdb68bba1cb37db029, scope.project.id=bad66f048c724ae380da0250ba68d56d, user.id=e65a84069e1d4cd7b0759a34dbfbe8dd) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:23.257181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:23.258702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:23.271662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 52/115] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:22 2022] GET /identity/v3/role_assignments?user.id=e65a84069e1d4cd7b0759a34dbfbe8dd&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=bcd685ff205446cdb68bba1cb37db029 => generated 259 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:23.272616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:23.276543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e12cde-9451-4d83-bfc1-07ffa2a60d86 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:23.362502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60bc13b-0cd2-4205-a581-a042946ced60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:23.363201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60bc13b-0cd2-4205-a581-a042946ced60 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:23.363487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60bc13b-0cd2-4205-a581-a042946ced60 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:23.363764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60bc13b-0cd2-4205-a581-a042946ced60 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:23.385037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60bc13b-0cd2-4205-a581-a042946ced60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-b60bc13b-0cd2-4205-a581-a042946ced60 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:23.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:23.391954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:23.393359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 64/116] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:26:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:23.399552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a72d7b4-0b88-4863-9e4e-095348993eba None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:23.399952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a72d7b4-0b88-4863-9e4e-095348993eba None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:23.400385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a72d7b4-0b88-4863-9e4e-095348993eba None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:23.402649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 65/117] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:23 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:23.407464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c61053-ae45-4fc8-976b-bf737dfd5c4f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:23.408278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a72d7b4-0b88-4863-9e4e-095348993eba None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:23.449899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb783a5-b8ee-4993-8bb3-90ed621b8503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:23.450559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb783a5-b8ee-4993-8bb3-90ed621b8503 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:23.450880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb783a5-b8ee-4993-8bb3-90ed621b8503 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:23.451159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb783a5-b8ee-4993-8bb3-90ed621b8503 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:23.463929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb783a5-b8ee-4993-8bb3-90ed621b8503 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-aeb783a5-b8ee-4993-8bb3-90ed621b8503 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:23.469538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:23.473101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 53/118] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:23.475951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32203a48-98ec-4f26-a0b0-0423c6afe83a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:23.486636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32203a48-98ec-4f26-a0b0-0423c6afe83a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:23.486636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32203a48-98ec-4f26-a0b0-0423c6afe83a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:23.496183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-32203a48-98ec-4f26-a0b0-0423c6afe83a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:23.497448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-32203a48-98ec-4f26-a0b0-0423c6afe83a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:23.517377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:23.575651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a72d7b4-0b88-4863-9e4e-095348993eba None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:23.586057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 54/119] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:23.596311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32203a48-98ec-4f26-a0b0-0423c6afe83a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:23.599297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:23.633836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:23.634576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:23.634907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:23.635265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:23.636143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:23.636597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:23.638166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:23.648641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 66/120] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:23.655438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:23.859892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17103c2-38d6-4514-9578-8734e33929ed None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:23.891623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32203a48-98ec-4f26-a0b0-0423c6afe83a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:23.901867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83d559c-7bd1-47b3-80bf-17b1151ea713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:23.902749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d559c-7bd1-47b3-80bf-17b1151ea713 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:23.903205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d559c-7bd1-47b3-80bf-17b1151ea713 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:23.903492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d559c-7bd1-47b3-80bf-17b1151ea713 None admin] PATH_INFO: `/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:23.904742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83d559c-7bd1-47b3-80bf-17b1151ea713 None admin] RBAC: Authorizing `identity:get_user(user_id=2f1e799f7d5b4182a2e0ecd338aaa415)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:23.906215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83d559c-7bd1-47b3-80bf-17b1151ea713 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:23.908041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 67/121] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:23 2022] GET /identity/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415 => generated 247 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:23.908854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83d559c-7bd1-47b3-80bf-17b1151ea713 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:23.974025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:23.975155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:23.975621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:23.976013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:23.976726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:23.977084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:23.979911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:23.990584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 55/122] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 404 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:23.991530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:24.002849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:24.016705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:24.084347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83d559c-7bd1-47b3-80bf-17b1151ea713 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:24.148242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0e976-764b-4f6b-9202-0b8b71622f1f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:24.164512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad3959d-c584-4ff0-8bca-9fa47696cb0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:24.165804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3959d-c584-4ff0-8bca-9fa47696cb0b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:24.166879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3959d-c584-4ff0-8bca-9fa47696cb0b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:24.167367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3959d-c584-4ff0-8bca-9fa47696cb0b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:24.190462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3c8016-e80c-47f0-8ca2-6d116268f313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:24.191361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3c8016-e80c-47f0-8ca2-6d116268f313 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:24.191750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3c8016-e80c-47f0-8ca2-6d116268f313 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:24.192124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3c8016-e80c-47f0-8ca2-6d116268f313 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:24.193416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3c8016-e80c-47f0-8ca2-6d116268f313 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:24.194462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad3959d-c584-4ff0-8bca-9fa47696cb0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:24.195168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3c8016-e80c-47f0-8ca2-6d116268f313 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:24.197717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 56/123] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:23 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:24.198632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3c8016-e80c-47f0-8ca2-6d116268f313 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5ad3959d-c584-4ff0-8bca-9fa47696cb0b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:24.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:24.202971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:24.203750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:24.204325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 68/124] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:26:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:24.205657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad3959d-c584-4ff0-8bca-9fa47696cb0b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:24.310118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:24.351484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3c8016-e80c-47f0-8ca2-6d116268f313 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:24.368504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad3959d-c584-4ff0-8bca-9fa47696cb0b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:24.372913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:24.408548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd70cb2-f40a-4d37-896c-113d0ff212ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:24.410064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd70cb2-f40a-4d37-896c-113d0ff212ba None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:24.411051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd70cb2-f40a-4d37-896c-113d0ff212ba None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:24.411497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd70cb2-f40a-4d37-896c-113d0ff212ba None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:24.425448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd70cb2-f40a-4d37-896c-113d0ff212ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:24.426356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143a47f1-b6b9-4acb-a4b5-469f8113e0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:24.427127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143a47f1-b6b9-4acb-a4b5-469f8113e0bd None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:24.427482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143a47f1-b6b9-4acb-a4b5-469f8113e0bd None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:24.427825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143a47f1-b6b9-4acb-a4b5-469f8113e0bd None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:24.429128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143a47f1-b6b9-4acb-a4b5-469f8113e0bd None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:24.431385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143a47f1-b6b9-4acb-a4b5-469f8113e0bd None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-afd70cb2-f40a-4d37-896c-113d0ff212ba None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:24.432693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:24.433639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:24.434415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:24.435049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 69/125] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:24 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:24.435049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143a47f1-b6b9-4acb-a4b5-469f8113e0bd None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:24.437206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 57/126] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:24.442944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd70cb2-f40a-4d37-896c-113d0ff212ba None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:24.606107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143a47f1-b6b9-4acb-a4b5-469f8113e0bd None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:24.618333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd70cb2-f40a-4d37-896c-113d0ff212ba None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:24.659156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bc7d63-3a10-4683-bd30-deb903110ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:24.660655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bc7d63-3a10-4683-bd30-deb903110ce0 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:24.661657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bc7d63-3a10-4683-bd30-deb903110ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:24.662402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bc7d63-3a10-4683-bd30-deb903110ce0 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d/users/2f1e799f7d5b4182a2e0ecd338aaa415/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:24.665455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bc7d63-3a10-4683-bd30-deb903110ce0 None admin] RBAC: Authorizing `identity:create_grant(project_id=bad66f048c724ae380da0250ba68d56d, user_id=2f1e799f7d5b4182a2e0ecd338aaa415, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:24.667220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bc7d63-3a10-4683-bd30-deb903110ce0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:24.670484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:24.671233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:24.671609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:24.671951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:24.672998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:24.672998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:24.674883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:24.684688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:24.694313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 58/127] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:26:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:24.694313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:24.696850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 70/128] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:24 2022] PUT /identity/v3/projects/bad66f048c724ae380da0250ba68d56d/users/2f1e799f7d5b4182a2e0ecd338aaa415/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:24.698166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bc7d63-3a10-4683-bd30-deb903110ce0 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:24.741445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:24.900845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bc7d63-3a10-4683-bd30-deb903110ce0 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:24.904417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e617391-22c8-4346-8df0-ec67734700a6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:24.986586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c589083-e7d3-431a-9117-0de76517ad84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:24.987533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c589083-e7d3-431a-9117-0de76517ad84 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:24.987963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c589083-e7d3-431a-9117-0de76517ad84 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:24.988402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c589083-e7d3-431a-9117-0de76517ad84 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:24.989854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c589083-e7d3-431a-9117-0de76517ad84 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:24.990486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:24.991452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:24.991867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:24.991994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c589083-e7d3-431a-9117-0de76517ad84 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:24.992306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:24.994092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:24.994671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:24.994940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 59/129] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:24 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:24.996723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c589083-e7d3-431a-9117-0de76517ad84 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:25.003347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:25.019253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 71/130] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:25.020213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:25.092503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:25.133600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c589083-e7d3-431a-9117-0de76517ad84 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:25.233718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:25.240702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f948f4-214a-4c95-a4a8-8db840d31a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:25.241822 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f948f4-214a-4c95-a4a8-8db840d31a85 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:25.242324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f948f4-214a-4c95-a4a8-8db840d31a85 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:25.243351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f948f4-214a-4c95-a4a8-8db840d31a85 None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:25.254089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9715a8c-3d3a-4a24-85bd-e91037567b1e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:25.257829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f948f4-214a-4c95-a4a8-8db840d31a85 None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:25.260945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f948f4-214a-4c95-a4a8-8db840d31a85 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:25.264268 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 60/131] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:24 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:25.265652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f948f4-214a-4c95-a4a8-8db840d31a85 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:25.364697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7928330d-7194-4ad1-9c93-5c71aae4eb3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:25.365392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7928330d-7194-4ad1-9c93-5c71aae4eb3f None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:25.365680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7928330d-7194-4ad1-9c93-5c71aae4eb3f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:25.365960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7928330d-7194-4ad1-9c93-5c71aae4eb3f None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d/users/e74c08a75a3a4d6fa1d1523a20e5fbb1/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:25.375690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7928330d-7194-4ad1-9c93-5c71aae4eb3f None admin] RBAC: Authorizing `identity:create_grant(project_id=bad66f048c724ae380da0250ba68d56d, user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:25.377965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7928330d-7194-4ad1-9c93-5c71aae4eb3f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:25.417797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 72/132] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:25 2022] PUT /identity/v3/projects/bad66f048c724ae380da0250ba68d56d/users/e74c08a75a3a4d6fa1d1523a20e5fbb1/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:25.421034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7928330d-7194-4ad1-9c93-5c71aae4eb3f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:25.560076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f948f4-214a-4c95-a4a8-8db840d31a85 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:25.630396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f23ebe8-528d-42e5-8a55-bd1ef8334a07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:25.631376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f23ebe8-528d-42e5-8a55-bd1ef8334a07 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:25.631659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f23ebe8-528d-42e5-8a55-bd1ef8334a07 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:25.631932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f23ebe8-528d-42e5-8a55-bd1ef8334a07 None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:25.635255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f23ebe8-528d-42e5-8a55-bd1ef8334a07 None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:25.636782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f23ebe8-528d-42e5-8a55-bd1ef8334a07 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:25.641960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 61/133] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:25 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 376 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:25.644133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626bf41b-1cc1-420b-b4a6-adf6e41ff2a9 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:25.646959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626bf41b-1cc1-420b-b4a6-adf6e41ff2a9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:25.647174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626bf41b-1cc1-420b-b4a6-adf6e41ff2a9 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:25.650915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 62/134] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:25 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:25.652435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626bf41b-1cc1-420b-b4a6-adf6e41ff2a9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:25.657269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:25.681338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7928330d-7194-4ad1-9c93-5c71aae4eb3f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:25.736773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666a5b9-dba5-4bc0-a998-cadc51437074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:25.738168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666a5b9-dba5-4bc0-a998-cadc51437074 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:25.738304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666a5b9-dba5-4bc0-a998-cadc51437074 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:25.738692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666a5b9-dba5-4bc0-a998-cadc51437074 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:25.740566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666a5b9-dba5-4bc0-a998-cadc51437074 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:25.743899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666a5b9-dba5-4bc0-a998-cadc51437074 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:25.747153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 73/135] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:25 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:25.749200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22487a-64e1-43e7-90fe-89442538a5f5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:25.749671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22487a-64e1-43e7-90fe-89442538a5f5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:25.750218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22487a-64e1-43e7-90fe-89442538a5f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:25.788037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-cb22487a-64e1-43e7-90fe-89442538a5f5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:25.789450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cb22487a-64e1-43e7-90fe-89442538a5f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:25.811402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626bf41b-1cc1-420b-b4a6-adf6e41ff2a9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:25.812541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:25.865419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982e458-b144-4500-a89a-61eca9e1bf90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:25.866637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982e458-b144-4500-a89a-61eca9e1bf90 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:25.867160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982e458-b144-4500-a89a-61eca9e1bf90 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:25.868579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982e458-b144-4500-a89a-61eca9e1bf90 None admin] PATH_INFO: `/v3/projects/adcafe4c8eaf49bda38a2fe27ad5bbf4` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:25.872606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982e458-b144-4500-a89a-61eca9e1bf90 None admin] RBAC: Authorizing `identity:get_project(project_id=adcafe4c8eaf49bda38a2fe27ad5bbf4)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:25.874371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982e458-b144-4500-a89a-61eca9e1bf90 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:25.877290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 63/136] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:25 2022] GET /identity/v3/projects/adcafe4c8eaf49bda38a2fe27ad5bbf4 => generated 312 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:25.878477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982e458-b144-4500-a89a-61eca9e1bf90 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:25.899630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 74/137] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:25.900658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb22487a-64e1-43e7-90fe-89442538a5f5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:25.993091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:26.019422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982e458-b144-4500-a89a-61eca9e1bf90 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:26.062646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3cead0-47ac-4f69-9595-51635d03e4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:26.063655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3cead0-47ac-4f69-9595-51635d03e4ab None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:26.064121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3cead0-47ac-4f69-9595-51635d03e4ab None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:26.064760 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3cead0-47ac-4f69-9595-51635d03e4ab None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d/users/e74c08a75a3a4d6fa1d1523a20e5fbb1/roles/f6e14365f5f54e2da132ac99f92451f5` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:26.067840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3cead0-47ac-4f69-9595-51635d03e4ab None admin] RBAC: Authorizing `identity:create_grant(project_id=bad66f048c724ae380da0250ba68d56d, user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1, role_id=f6e14365f5f54e2da132ac99f92451f5)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:26.069642 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3cead0-47ac-4f69-9595-51635d03e4ab None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:26.090236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 64/138] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:25 2022] PUT /identity/v3/projects/bad66f048c724ae380da0250ba68d56d/users/e74c08a75a3a4d6fa1d1523a20e5fbb1/roles/f6e14365f5f54e2da132ac99f92451f5 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:26.091986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea58fb0-6a21-48dc-85f2-75f2baf1ea06 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:26.092279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea58fb0-6a21-48dc-85f2-75f2baf1ea06 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:26.092558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea58fb0-6a21-48dc-85f2-75f2baf1ea06 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:26.094123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 65/139] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:26.100980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4076e9-069f-4dd2-aff0-93a000520f31 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:26.101432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4076e9-069f-4dd2-aff0-93a000520f31 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:26.101889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4076e9-069f-4dd2-aff0-93a000520f31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:26.140742 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-cf4076e9-069f-4dd2-aff0-93a000520f31 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:26.143059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-cf4076e9-069f-4dd2-aff0-93a000520f31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:26.170950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb22487a-64e1-43e7-90fe-89442538a5f5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:26.260268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7dda1a-8b59-491b-ab22-e13917619728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:26.260965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7dda1a-8b59-491b-ab22-e13917619728 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:26.261317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7dda1a-8b59-491b-ab22-e13917619728 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:26.261515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7dda1a-8b59-491b-ab22-e13917619728 None admin] PATH_INFO: `/v3/projects/adcafe4c8eaf49bda38a2fe27ad5bbf4/users/e74c08a75a3a4d6fa1d1523a20e5fbb1/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:26.266461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7dda1a-8b59-491b-ab22-e13917619728 None admin] RBAC: Authorizing `identity:create_grant(project_id=adcafe4c8eaf49bda38a2fe27ad5bbf4, user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:26.267992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7dda1a-8b59-491b-ab22-e13917619728 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:26.301042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 75/140] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:25 2022] PUT /identity/v3/projects/adcafe4c8eaf49bda38a2fe27ad5bbf4/users/e74c08a75a3a4d6fa1d1523a20e5fbb1/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:26.330190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 66/141] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:26.431137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7dda1a-8b59-491b-ab22-e13917619728 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:26.452468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:26.573208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7dda1a-8b59-491b-ab22-e13917619728 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:26.626484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8ca493-63f3-4cbe-8f98-871488166889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:26.627112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8ca493-63f3-4cbe-8f98-871488166889 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:26.627463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8ca493-63f3-4cbe-8f98-871488166889 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:26.627757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8ca493-63f3-4cbe-8f98-871488166889 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:26.629190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8ca493-63f3-4cbe-8f98-871488166889 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:26.630798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8ca493-63f3-4cbe-8f98-871488166889 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:26.632444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 76/142] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:26.638546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4076e9-069f-4dd2-aff0-93a000520f31 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:26.740281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:26.818188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4076e9-069f-4dd2-aff0-93a000520f31 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:26.860503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8ca493-63f3-4cbe-8f98-871488166889 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:26.883557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950efa97-ec3b-4ed2-b56a-2e18a9a7d792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:26.884080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950efa97-ec3b-4ed2-b56a-2e18a9a7d792 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:26.884363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950efa97-ec3b-4ed2-b56a-2e18a9a7d792 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:26.884642 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950efa97-ec3b-4ed2-b56a-2e18a9a7d792 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:26.885391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950efa97-ec3b-4ed2-b56a-2e18a9a7d792 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:26.886923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950efa97-ec3b-4ed2-b56a-2e18a9a7d792 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:26.929490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 67/143] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:26:26 2022] POST /identity/v3/users => generated 283 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:26:27.030827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:27.061539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8ca493-63f3-4cbe-8f98-871488166889 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:27.116690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dbe8b1-a013-4be2-b4ee-98986069b689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:27.117407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dbe8b1-a013-4be2-b4ee-98986069b689 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:27.117803 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dbe8b1-a013-4be2-b4ee-98986069b689 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:27.118089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dbe8b1-a013-4be2-b4ee-98986069b689 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:27.143514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dbe8b1-a013-4be2-b4ee-98986069b689 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-e4dbe8b1-a013-4be2-b4ee-98986069b689 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:27.152748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:27.153667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:27.154882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 77/144] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:27.160022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950efa97-ec3b-4ed2-b56a-2e18a9a7d792 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:27.325527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950efa97-ec3b-4ed2-b56a-2e18a9a7d792 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:27.351322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:27.373320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:27.374073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:27.374418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:27.374780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:27.375981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:27.376506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:27.379001 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:27.396828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 68/145] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:27.402912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dbe8b1-a013-4be2-b4ee-98986069b689 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:27.559451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dbe8b1-a013-4be2-b4ee-98986069b689 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:27.616728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f2473f-15da-434a-a391-86eb28b34b6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:27.617392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f2473f-15da-434a-a391-86eb28b34b6e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:27.618215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f2473f-15da-434a-a391-86eb28b34b6e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:27.618567 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f2473f-15da-434a-a391-86eb28b34b6e None admin] PATH_INFO: `/v3/users/e65a84069e1d4cd7b0759a34dbfbe8dd` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:27.620076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f2473f-15da-434a-a391-86eb28b34b6e None admin] RBAC: Authorizing `identity:get_user(user_id=e65a84069e1d4cd7b0759a34dbfbe8dd)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:27.621575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f2473f-15da-434a-a391-86eb28b34b6e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:27.623324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 78/146] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:27 2022] GET /identity/v3/users/e65a84069e1d4cd7b0759a34dbfbe8dd => generated 289 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:27.629102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:27.758714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883be207-34ce-48e8-9f21-31c89c7e1ea2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:27.810090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6826da2-cd3f-4d50-a5b9-376ad21f3942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:27.810792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6826da2-cd3f-4d50-a5b9-376ad21f3942 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:27.811164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6826da2-cd3f-4d50-a5b9-376ad21f3942 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:27.811519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6826da2-cd3f-4d50-a5b9-376ad21f3942 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:27.812950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6826da2-cd3f-4d50-a5b9-376ad21f3942 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:27.814462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6826da2-cd3f-4d50-a5b9-376ad21f3942 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:27.821329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 69/147] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:27 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:27.827511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f2473f-15da-434a-a391-86eb28b34b6e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:27.840548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:27.873083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276a6406-e67e-49aa-87ad-65408d04dfe0 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:27.873403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276a6406-e67e-49aa-87ad-65408d04dfe0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:27.873640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276a6406-e67e-49aa-87ad-65408d04dfe0 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:27.875282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 70/148] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:27.880680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8bfe59-fef3-46a3-8fc3-341dc2ec77e9 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:27.880958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8bfe59-fef3-46a3-8fc3-341dc2ec77e9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:27.881233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8bfe59-fef3-46a3-8fc3-341dc2ec77e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:27.904287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-fb8bfe59-fef3-46a3-8fc3-341dc2ec77e9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:27.905863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fb8bfe59-fef3-46a3-8fc3-341dc2ec77e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:27.983131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f2473f-15da-434a-a391-86eb28b34b6e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:27.985322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:28.018273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 71/149] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:28.041860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa2e7a-2d06-421c-a8f9-b5148064a0a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:28.042828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa2e7a-2d06-421c-a8f9-b5148064a0a7 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:28.043130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa2e7a-2d06-421c-a8f9-b5148064a0a7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:28.043536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa2e7a-2d06-421c-a8f9-b5148064a0a7 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d/users/e65a84069e1d4cd7b0759a34dbfbe8dd/roles/bcd685ff205446cdb68bba1cb37db029` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:28.046376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa2e7a-2d06-421c-a8f9-b5148064a0a7 None admin] RBAC: Authorizing `identity:create_grant(project_id=bad66f048c724ae380da0250ba68d56d, user_id=e65a84069e1d4cd7b0759a34dbfbe8dd, role_id=bcd685ff205446cdb68bba1cb37db029)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:28.047903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa2e7a-2d06-421c-a8f9-b5148064a0a7 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:28.068127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 79/150] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:27 2022] PUT /identity/v3/projects/bad66f048c724ae380da0250ba68d56d/users/e65a84069e1d4cd7b0759a34dbfbe8dd/roles/bcd685ff205446cdb68bba1cb37db029 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:28.386194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8bfe59-fef3-46a3-8fc3-341dc2ec77e9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:28.510140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8bfe59-fef3-46a3-8fc3-341dc2ec77e9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:28.550692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb39e5-7ad0-406a-9464-b8a4e37947e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:28.551718 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb39e5-7ad0-406a-9464-b8a4e37947e7 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:28.552329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb39e5-7ad0-406a-9464-b8a4e37947e7 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:28.552820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb39e5-7ad0-406a-9464-b8a4e37947e7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:28.568109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cb39e5-7ad0-406a-9464-b8a4e37947e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-06cb39e5-7ad0-406a-9464-b8a4e37947e7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:28.574546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:28.575941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:28.576982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:28.579235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 72/151] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:26:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:28.585839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa2e7a-2d06-421c-a8f9-b5148064a0a7 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:28.719368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:28.775222 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa2e7a-2d06-421c-a8f9-b5148064a0a7 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:28.830324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:28.831141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:28.831518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:28.831849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:28.832593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:28.832901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:28.834348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:28.846133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 80/152] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:28.855276 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb39e5-7ad0-406a-9464-b8a4e37947e7 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:28.940120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:29.035134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb39e5-7ad0-406a-9464-b8a4e37947e7 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:29.103688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216f20f2-cd23-4fe9-9e5b-0206b66e458e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:29.104607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f20f2-cd23-4fe9-9e5b-0206b66e458e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:29.105075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f20f2-cd23-4fe9-9e5b-0206b66e458e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:29.105565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216f20f2-cd23-4fe9-9e5b-0206b66e458e None admin] PATH_INFO: `/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:29.107772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216f20f2-cd23-4fe9-9e5b-0206b66e458e None admin] RBAC: Authorizing `identity:get_user(user_id=2f1e799f7d5b4182a2e0ecd338aaa415)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:29.109848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216f20f2-cd23-4fe9-9e5b-0206b66e458e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:29.112020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 73/153] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:28 2022] GET /identity/v3/users/2f1e799f7d5b4182a2e0ecd338aaa415 => generated 247 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:29.118507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:29.253502 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51fbdd-da51-4273-8a28-0643f5607666 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:29.253928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51fbdd-da51-4273-8a28-0643f5607666 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:29.254583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51fbdd-da51-4273-8a28-0643f5607666 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:29.257191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 74/154] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:29.263043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e119b1-45c1-460f-982c-b4ee9980e390 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:29.263685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e119b1-45c1-460f-982c-b4ee9980e390 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:29.264293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e119b1-45c1-460f-982c-b4ee9980e390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:29.274781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e06cda-c1be-4080-b25a-2ef6e15f66e9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:29.294364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e4e119b1-45c1-460f-982c-b4ee9980e390 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:29.296109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e4e119b1-45c1-460f-982c-b4ee9980e390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:29.343597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:29.365711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b18d00-9428-45d6-98f6-f133e46b71c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:29.367067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b18d00-9428-45d6-98f6-f133e46b71c4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:29.367827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b18d00-9428-45d6-98f6-f133e46b71c4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:29.368997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b18d00-9428-45d6-98f6-f133e46b71c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:29.420284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 75/155] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:29.532143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b18d00-9428-45d6-98f6-f133e46b71c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:29.534797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b18d00-9428-45d6-98f6-f133e46b71c4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:29.676252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 81/156] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 559 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:26:29.682428 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e119b1-45c1-460f-982c-b4ee9980e390 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:29.745964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:29.834680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e119b1-45c1-460f-982c-b4ee9980e390 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:29.872108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57276bef-45ee-4ae2-af25-b20d42116399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:29.873110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57276bef-45ee-4ae2-af25-b20d42116399 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:29.873110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57276bef-45ee-4ae2-af25-b20d42116399 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:29.873380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57276bef-45ee-4ae2-af25-b20d42116399 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:29.874632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57276bef-45ee-4ae2-af25-b20d42116399 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:29.876154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57276bef-45ee-4ae2-af25-b20d42116399 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:29.878100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 76/157] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:29 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:29.887870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b18d00-9428-45d6-98f6-f133e46b71c4 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:29.972687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd3c72-1bdc-4d80-ac10-95eef7d45ddc None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:29.973207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd3c72-1bdc-4d80-ac10-95eef7d45ddc None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:29.973633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd3c72-1bdc-4d80-ac10-95eef7d45ddc None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:29.975439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 77/158] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:29.979460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:30.009290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0333a6ae-3994-4f35-8c80-7758d3e9bcb1 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:30.010018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0333a6ae-3994-4f35-8c80-7758d3e9bcb1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:30.010528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0333a6ae-3994-4f35-8c80-7758d3e9bcb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:30.043705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-0333a6ae-3994-4f35-8c80-7758d3e9bcb1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:30.045264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0333a6ae-3994-4f35-8c80-7758d3e9bcb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:30.113104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b18d00-9428-45d6-98f6-f133e46b71c4 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:30.146659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 78/159] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:30.179102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:30.180095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:30.180609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:30.181094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:30.182448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=bad66f048c724ae380da0250ba68d56d, user.id=2f1e799f7d5b4182a2e0ecd338aaa415) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:30.183027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:30.185763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:30.197962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 82/160] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:29 2022] GET /identity/v3/role_assignments?user.id=2f1e799f7d5b4182a2e0ecd338aaa415&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=38171c89968943bd92d9e9758db65615 => generated 608 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:30.263178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0333a6ae-3994-4f35-8c80-7758d3e9bcb1 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:30.317163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:30.408186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0333a6ae-3994-4f35-8c80-7758d3e9bcb1 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:30.457802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199ac00f-1bc5-4664-ba45-4d6281e130ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:30.458703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ac00f-1bc5-4664-ba45-4d6281e130ce None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:30.459147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ac00f-1bc5-4664-ba45-4d6281e130ce None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:30.459617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ac00f-1bc5-4664-ba45-4d6281e130ce None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:30.473434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199ac00f-1bc5-4664-ba45-4d6281e130ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-199ac00f-1bc5-4664-ba45-4d6281e130ce None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:30.483852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:30.484932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:30.485901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:30.488285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 79/161] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:30.495675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:30.670811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:30.685555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c941e4e8-bbf3-4586-849c-be503192ba97 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:30.720118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199ac00f-1bc5-4664-ba45-4d6281e130ce None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:30.744076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:30.744774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:30.745066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:30.745347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:30.746056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:30.746368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:30.747982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:30.763294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 83/162] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:30.772153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:31.031505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f5a5a1-208b-43dd-9c61-93fde7c03927 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:31.047113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:31.075785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00246e79-a08f-4906-b488-417bac4c8048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:31.076791 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00246e79-a08f-4906-b488-417bac4c8048 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:31.077287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00246e79-a08f-4906-b488-417bac4c8048 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:31.077779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00246e79-a08f-4906-b488-417bac4c8048 None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:31.078488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199ac00f-1bc5-4664-ba45-4d6281e130ce None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:31.079423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00246e79-a08f-4906-b488-417bac4c8048 None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:31.081090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00246e79-a08f-4906-b488-417bac4c8048 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:31.083196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 84/163] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:30 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:31.089863 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00246e79-a08f-4906-b488-417bac4c8048 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:31.130315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:31.221492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a02b5-ba90-432a-99d4-d62d58a613ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:31.222549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a02b5-ba90-432a-99d4-d62d58a613ae None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:31.223274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a02b5-ba90-432a-99d4-d62d58a613ae None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:31.223937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a02b5-ba90-432a-99d4-d62d58a613ae None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:31.233035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00246e79-a08f-4906-b488-417bac4c8048 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:31.256273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438a02b5-ba90-432a-99d4-d62d58a613ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-438a02b5-ba90-432a-99d4-d62d58a613ae None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:31.266703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:31.267703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:31.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:31.272829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 80/164] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:26:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 553 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:26:31.285908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b00019-c44f-4c90-8ddb-8c8bc87497e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:31.286091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a02b5-ba90-432a-99d4-d62d58a613ae None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:31.287227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b00019-c44f-4c90-8ddb-8c8bc87497e1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:31.287815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b00019-c44f-4c90-8ddb-8c8bc87497e1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:31.288278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b00019-c44f-4c90-8ddb-8c8bc87497e1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:31.377252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b00019-c44f-4c90-8ddb-8c8bc87497e1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:31.379266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b00019-c44f-4c90-8ddb-8c8bc87497e1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:31.410752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:31.516514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 85/165] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:26:31.527106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b00019-c44f-4c90-8ddb-8c8bc87497e1 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:31.693830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a02b5-ba90-432a-99d4-d62d58a613ae None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:31.747952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b00019-c44f-4c90-8ddb-8c8bc87497e1 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:31.804956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:31.828171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:31.828821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:31.829102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:31.829374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:31.830067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:31.830369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:31.831925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:31.833613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8890f6-57e4-4fe3-90ac-854522baa46e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:31.834274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8890f6-57e4-4fe3-90ac-854522baa46e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:31.834622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8890f6-57e4-4fe3-90ac-854522baa46e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:31.834966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8890f6-57e4-4fe3-90ac-854522baa46e None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:31.839657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8890f6-57e4-4fe3-90ac-854522baa46e None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:31.841182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8890f6-57e4-4fe3-90ac-854522baa46e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:31.843278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 86/166] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:31 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:31.849510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8890f6-57e4-4fe3-90ac-854522baa46e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:31.856773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 81/167] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:26:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 572 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:31.867832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:31.910758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:32.108703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8890f6-57e4-4fe3-90ac-854522baa46e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:32.191231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb151c1-f3c9-4504-9cf1-e242fec74171 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:32.193021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbbdabc-8fcb-4bd8-8286-a1a49b2d28d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:32.194111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbbdabc-8fcb-4bd8-8286-a1a49b2d28d1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:32.194603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbbdabc-8fcb-4bd8-8286-a1a49b2d28d1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:32.195093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbbdabc-8fcb-4bd8-8286-a1a49b2d28d1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:32.196630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbbdabc-8fcb-4bd8-8286-a1a49b2d28d1 None admin] RBAC: Adding query filter params (role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=bad66f048c724ae380da0250ba68d56d, user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:32.197170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbbdabc-8fcb-4bd8-8286-a1a49b2d28d1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:32.198876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbbdabc-8fcb-4bd8-8286-a1a49b2d28d1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:32.212571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 87/168] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:31 2022] GET /identity/v3/role_assignments?user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=0eb5172435534daf81351f9eb5281719 => generated 608 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:32.214218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6db1e-2bfe-4a39-9900-6e7d9bbe03a9 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:32.215375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6db1e-2bfe-4a39-9900-6e7d9bbe03a9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:32.215902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6db1e-2bfe-4a39-9900-6e7d9bbe03a9 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:32.217571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 88/169] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:32.237492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef11c8-30e1-4860-9983-90dcb1fbccb8 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:32.238552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef11c8-30e1-4860-9983-90dcb1fbccb8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:32.239322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef11c8-30e1-4860-9983-90dcb1fbccb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:32.284226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-dcef11c8-30e1-4860-9983-90dcb1fbccb8 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:32.286420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dcef11c8-30e1-4860-9983-90dcb1fbccb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:32.341356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4844b6-b33c-43c4-bf69-3a0269109ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:32.342443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4844b6-b33c-43c4-bf69-3a0269109ccf None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:32.342942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4844b6-b33c-43c4-bf69-3a0269109ccf None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:32.343635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4844b6-b33c-43c4-bf69-3a0269109ccf None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:32.346069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4844b6-b33c-43c4-bf69-3a0269109ccf None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:32.353220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4844b6-b33c-43c4-bf69-3a0269109ccf None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:32.361197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 82/170] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:31 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 494 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:32.367413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4844b6-b33c-43c4-bf69-3a0269109ccf None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:32.450258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:32.489833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 89/171] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:32.613306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:32.640772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4844b6-b33c-43c4-bf69-3a0269109ccf None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:32.695747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56d240b-14c4-4894-a966-da4828609fa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:32.697268 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d240b-14c4-4894-a966-da4828609fa4 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:32.697620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d240b-14c4-4894-a966-da4828609fa4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:32.697888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d240b-14c4-4894-a966-da4828609fa4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:32.800543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56d240b-14c4-4894-a966-da4828609fa4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:32.802899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56d240b-14c4-4894-a966-da4828609fa4 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:32.931730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:32.951793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 83/172] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 585 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:32.958337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcef11c8-30e1-4860-9983-90dcb1fbccb8 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:33.196498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daef49c-9924-49c7-992c-a4ba454d9419 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.197252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daef49c-9924-49c7-992c-a4ba454d9419 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.197252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daef49c-9924-49c7-992c-a4ba454d9419 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.199749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 84/173] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:33.205506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b0da6-8782-4022-bd01-164538857b18 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.205886 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b0da6-8782-4022-bd01-164538857b18 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.206033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b0da6-8782-4022-bd01-164538857b18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.241148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcef11c8-30e1-4860-9983-90dcb1fbccb8 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:33.247800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-246b0da6-8782-4022-bd01-164538857b18 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:33.249025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-246b0da6-8782-4022-bd01-164538857b18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:33.303842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13309389-d951-4e92-9a6b-c58da191c81b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:33.304492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13309389-d951-4e92-9a6b-c58da191c81b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.304837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13309389-d951-4e92-9a6b-c58da191c81b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.305107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13309389-d951-4e92-9a6b-c58da191c81b None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.306849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13309389-d951-4e92-9a6b-c58da191c81b None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:33.308370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13309389-d951-4e92-9a6b-c58da191c81b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:33.310359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 90/174] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:32 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:33.311233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13309389-d951-4e92-9a6b-c58da191c81b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:33.351432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 85/175] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:33.353376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e379e67d-84eb-4c2b-a68c-9176a175e1ea None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.353655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e379e67d-84eb-4c2b-a68c-9176a175e1ea None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.353925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e379e67d-84eb-4c2b-a68c-9176a175e1ea None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.363756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 86/176] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:33 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:33.364705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e379e67d-84eb-4c2b-a68c-9176a175e1ea None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:33.411273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:33.478839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13309389-d951-4e92-9a6b-c58da191c81b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:33.493362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e379e67d-84eb-4c2b-a68c-9176a175e1ea None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:33.508976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:33.521893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c07c22-2a0e-4447-8746-ab0f0c682488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:33.522795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c07c22-2a0e-4447-8746-ab0f0c682488 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.523366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c07c22-2a0e-4447-8746-ab0f0c682488 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.523746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c07c22-2a0e-4447-8746-ab0f0c682488 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.537310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c07c22-2a0e-4447-8746-ab0f0c682488 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-79c07c22-2a0e-4447-8746-ab0f0c682488 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:33.543200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:33.544271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:33.545154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 91/177] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:33.545799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002fce4c-e4a9-471e-804d-22ce8eed2096 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.546051 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002fce4c-e4a9-471e-804d-22ce8eed2096 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.546325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002fce4c-e4a9-471e-804d-22ce8eed2096 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.548202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c09eb0b-fcc6-4868-94a9-79a7c1830b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:33.549018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c09eb0b-fcc6-4868-94a9-79a7c1830b88 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.549494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c09eb0b-fcc6-4868-94a9-79a7c1830b88 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.549940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c09eb0b-fcc6-4868-94a9-79a7c1830b88 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.550879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c09eb0b-fcc6-4868-94a9-79a7c1830b88 None admin] RBAC: Adding query filter params (role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=bad66f048c724ae380da0250ba68d56d, user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:33.551323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c09eb0b-fcc6-4868-94a9-79a7c1830b88 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:33.553080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c09eb0b-fcc6-4868-94a9-79a7c1830b88 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:33.566773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 87/178] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:33 2022] GET /identity/v3/role_assignments?user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:33.566978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-002fce4c-e4a9-471e-804d-22ce8eed2096 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:33.567772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-002fce4c-e4a9-471e-804d-22ce8eed2096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:33.568815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef14229-3843-4008-b4fb-968e3ea49a64 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.569290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef14229-3843-4008-b4fb-968e3ea49a64 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.569699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef14229-3843-4008-b4fb-968e3ea49a64 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.571305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 88/179] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:33.572204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef14229-3843-4008-b4fb-968e3ea49a64 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:33.668555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 92/180] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:33.669645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88a0960-f87f-435b-b998-b51a42cfeeab None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.669932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88a0960-f87f-435b-b998-b51a42cfeeab None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.670204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88a0960-f87f-435b-b998-b51a42cfeeab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.699478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-b88a0960-f87f-435b-b998-b51a42cfeeab None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:33.701222 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b88a0960-f87f-435b-b998-b51a42cfeeab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:33.725183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef14229-3843-4008-b4fb-968e3ea49a64 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:33.766195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:33.766926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:33.767499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.767950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.768408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.769353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:33.769846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:33.771594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:33.782438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 89/181] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:33.788812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:33.812677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 93/182] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:33.814201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd668796-4a71-4410-9216-f7444f36f754 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.817323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd668796-4a71-4410-9216-f7444f36f754 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.817323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd668796-4a71-4410-9216-f7444f36f754 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.817323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 94/183] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:33.819715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd668796-4a71-4410-9216-f7444f36f754 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:33.845773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:33.909804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac092ee2-0647-476e-b705-4b85b15e2a83 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:33.952162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd504c96-7cca-4bb9-9209-0e1093be7d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:33.953340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd504c96-7cca-4bb9-9209-0e1093be7d49 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.953821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd504c96-7cca-4bb9-9209-0e1093be7d49 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.954314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd504c96-7cca-4bb9-9209-0e1093be7d49 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:33.961861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd668796-4a71-4410-9216-f7444f36f754 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:33.967766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd504c96-7cca-4bb9-9209-0e1093be7d49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-bd504c96-7cca-4bb9-9209-0e1093be7d49 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:33.973617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:33.974583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:33.975418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:33.978121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 90/184] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:26:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:33.979664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b07b2-0738-4203-a023-ccb9b290c51a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:33.980189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b07b2-0738-4203-a023-ccb9b290c51a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:33.980670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b07b2-0738-4203-a023-ccb9b290c51a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.011897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-442b07b2-0738-4203-a023-ccb9b290c51a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:34.013317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-442b07b2-0738-4203-a023-ccb9b290c51a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:34.019500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f7a97-1083-465e-9ca1-b0a5ada0b58e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:34.022212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f7a97-1083-465e-9ca1-b0a5ada0b58e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:34.022212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f7a97-1083-465e-9ca1-b0a5ada0b58e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:34.022212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f7a97-1083-465e-9ca1-b0a5ada0b58e None admin] PATH_INFO: `/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.023907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687f7a97-1083-465e-9ca1-b0a5ada0b58e None admin] RBAC: Authorizing `identity:get_user(user_id=e74c08a75a3a4d6fa1d1523a20e5fbb1)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:34.026512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687f7a97-1083-465e-9ca1-b0a5ada0b58e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:34.029324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 95/185] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:33 2022] GET /identity/v3/users/e74c08a75a3a4d6fa1d1523a20e5fbb1 => generated 275 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:34.030004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f7a97-1083-465e-9ca1-b0a5ada0b58e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:34.120463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:34.142787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 91/186] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:34.142787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b07b2-0738-4203-a023-ccb9b290c51a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:34.208437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:34.240880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f7a97-1083-465e-9ca1-b0a5ada0b58e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:34.278577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:34.279509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:34.279509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:34.279752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.280314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:34.280647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:34.282140 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:34.292470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 96/187] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:34.293358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:34.328152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b07b2-0738-4203-a023-ccb9b290c51a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:34.377529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ff2e58-54f3-466a-924f-6d2c3db55adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:34.378202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff2e58-54f3-466a-924f-6d2c3db55adb None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:34.378487 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff2e58-54f3-466a-924f-6d2c3db55adb None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:34.378786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff2e58-54f3-466a-924f-6d2c3db55adb None admin] PATH_INFO: `/v3/domains/default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.380033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ff2e58-54f3-466a-924f-6d2c3db55adb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:34.382435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ff2e58-54f3-466a-924f-6d2c3db55adb None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:34.383162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 92/188] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:34.388590 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ff2e58-54f3-466a-924f-6d2c3db55adb None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:34.439180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a3950f-2e6f-4886-9028-d6c1bca6a9fb None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:34.441927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:34.499464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30da7ea4-ec62-4488-8258-3d0358be054e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:34.500752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30da7ea4-ec62-4488-8258-3d0358be054e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:34.501184 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30da7ea4-ec62-4488-8258-3d0358be054e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:34.501607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30da7ea4-ec62-4488-8258-3d0358be054e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.520867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:34.525219 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ff2e58-54f3-466a-924f-6d2c3db55adb None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:34.579541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e477f-e866-4bc5-bfee-a67bb3b3d80e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:34.580125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e477f-e866-4bc5-bfee-a67bb3b3d80e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:34.580388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e477f-e866-4bc5-bfee-a67bb3b3d80e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:34.580665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e477f-e866-4bc5-bfee-a67bb3b3d80e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.591248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30da7ea4-ec62-4488-8258-3d0358be054e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:34.592263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5e477f-e866-4bc5-bfee-a67bb3b3d80e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:34.592776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30da7ea4-ec62-4488-8258-3d0358be054e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2f5e477f-e866-4bc5-bfee-a67bb3b3d80e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:34.597557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:34.598559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:34.599404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 93/189] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:34.601597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e477f-e866-4bc5-bfee-a67bb3b3d80e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:34.720273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 97/190] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:34.721089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e477f-e866-4bc5-bfee-a67bb3b3d80e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:34.721250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30da7ea4-ec62-4488-8258-3d0358be054e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:34.725013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:34.765165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f63d90-27d4-4a11-ba4e-d859612a82c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:34.765796 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f63d90-27d4-4a11-ba4e-d859612a82c0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:34.766070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f63d90-27d4-4a11-ba4e-d859612a82c0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:34.766339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f63d90-27d4-4a11-ba4e-d859612a82c0 None admin] PATH_INFO: `/v3/users/9d54392a601648ee99905a3f024dd0be` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.788611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f63d90-27d4-4a11-ba4e-d859612a82c0 None admin] RBAC: Authorizing `identity:get_user(user_id=9d54392a601648ee99905a3f024dd0be)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:34.791204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f63d90-27d4-4a11-ba4e-d859612a82c0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:34.793467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 94/191] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:34 2022] GET /identity/v3/users/9d54392a601648ee99905a3f024dd0be => generated 283 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:34.794369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f63d90-27d4-4a11-ba4e-d859612a82c0 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:34.820043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:34.869016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30da7ea4-ec62-4488-8258-3d0358be054e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:34.911558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f63d90-27d4-4a11-ba4e-d859612a82c0 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:34.916114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b267a4-53c5-41da-a94f-ee88cbeb7b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:34.917202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b267a4-53c5-41da-a94f-ee88cbeb7b74 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:34.917696 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b267a4-53c5-41da-a94f-ee88cbeb7b74 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:34.918227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b267a4-53c5-41da-a94f-ee88cbeb7b74 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.919237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b267a4-53c5-41da-a94f-ee88cbeb7b74 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:34.920735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b267a4-53c5-41da-a94f-ee88cbeb7b74 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:34.946821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 98/192] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:26:34 2022] POST /identity/v3/users => generated 297 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:34.947679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b267a4-53c5-41da-a94f-ee88cbeb7b74 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:34.961323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482dde0f-308e-4010-ab4f-e3033314704c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:34.962669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482dde0f-308e-4010-ab4f-e3033314704c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:34.963102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482dde0f-308e-4010-ab4f-e3033314704c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:34.963367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482dde0f-308e-4010-ab4f-e3033314704c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:34.987839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482dde0f-308e-4010-ab4f-e3033314704c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-482dde0f-308e-4010-ab4f-e3033314704c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:34.993145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:34.994477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:34.995396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:35.002855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 95/193] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:26:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:35.003979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482dde0f-308e-4010-ab4f-e3033314704c None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:35.059994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:35.123534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:35.130595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b267a4-53c5-41da-a94f-ee88cbeb7b74 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:35.165756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482dde0f-308e-4010-ab4f-e3033314704c None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:35.176790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:35.177523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:35.177809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:35.178086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:35.178842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] RBAC: Adding query filter params (name=reader) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:35.179158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:35.180596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:35.190135 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 99/194] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:34 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:35.191058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:35.220522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9ac40b-a52c-4708-8f55-d1d25eb3ec00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:35.221333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9ac40b-a52c-4708-8f55-d1d25eb3ec00 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:35.221333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9ac40b-a52c-4708-8f55-d1d25eb3ec00 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:35.221606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9ac40b-a52c-4708-8f55-d1d25eb3ec00 None admin] PATH_INFO: `/v3/projects/adcafe4c8eaf49bda38a2fe27ad5bbf4` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:35.223445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9ac40b-a52c-4708-8f55-d1d25eb3ec00 None admin] RBAC: Authorizing `identity:get_project(project_id=adcafe4c8eaf49bda38a2fe27ad5bbf4)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:35.225825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9ac40b-a52c-4708-8f55-d1d25eb3ec00 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:35.228295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 96/195] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:35 2022] GET /identity/v3/projects/adcafe4c8eaf49bda38a2fe27ad5bbf4 => generated 312 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:35.231663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9ac40b-a52c-4708-8f55-d1d25eb3ec00 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:35.364939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7e1da9-264c-460a-abff-23b2156c71dc None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:35.400045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:35.417014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9ac40b-a52c-4708-8f55-d1d25eb3ec00 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:35.422207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af4e866-123c-417e-bfdc-80ae048ac963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:35.423498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4e866-123c-417e-bfdc-80ae048ac963 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:35.423990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4e866-123c-417e-bfdc-80ae048ac963 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:35.424500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4e866-123c-417e-bfdc-80ae048ac963 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:35.489398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:35.491995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:35.494120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:35.494924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:35.496823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:35.497615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:35.499333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:35.510856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 97/196] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:26:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:35.512072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:35.524929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:35.545075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af4e866-123c-417e-bfdc-80ae048ac963 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:35.548515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af4e866-123c-417e-bfdc-80ae048ac963 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:35.687816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 100/197] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:35.689109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af4e866-123c-417e-bfdc-80ae048ac963 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:35.701328 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc5254-8118-4ec3-ad62-7cdd7733e17a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:35.717029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:35.767821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437b356d-dc34-4181-81bd-a950b3971a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:35.769003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437b356d-dc34-4181-81bd-a950b3971a02 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:35.769519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437b356d-dc34-4181-81bd-a950b3971a02 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:35.770021 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437b356d-dc34-4181-81bd-a950b3971a02 None admin] PATH_INFO: `/v3/users/e65a84069e1d4cd7b0759a34dbfbe8dd` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:35.772757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437b356d-dc34-4181-81bd-a950b3971a02 None admin] RBAC: Authorizing `identity:get_user(user_id=e65a84069e1d4cd7b0759a34dbfbe8dd)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:35.775896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437b356d-dc34-4181-81bd-a950b3971a02 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:35.779336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 98/198] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:35 2022] GET /identity/v3/users/e65a84069e1d4cd7b0759a34dbfbe8dd => generated 289 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:35.780829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437b356d-dc34-4181-81bd-a950b3971a02 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:35.818925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af4e866-123c-417e-bfdc-80ae048ac963 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:35.910777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:35.911838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:35.912350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:35.912816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:35.913768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] RBAC: Adding query filter params (role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=adcafe4c8eaf49bda38a2fe27ad5bbf4, user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:35.914279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:35.916711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:35.926487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 101/199] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:35 2022] GET /identity/v3/role_assignments?user.id=e74c08a75a3a4d6fa1d1523a20e5fbb1&scope.project.id=adcafe4c8eaf49bda38a2fe27ad5bbf4&role.id=0eb5172435534daf81351f9eb5281719 => generated 608 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:35.927851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:35.939113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:36.015938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437b356d-dc34-4181-81bd-a950b3971a02 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:36.069449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e97fc1-d779-4b41-87e1-674b6cd98cb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:36.070500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e97fc1-d779-4b41-87e1-674b6cd98cb5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:36.071018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e97fc1-d779-4b41-87e1-674b6cd98cb5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:36.072355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e97fc1-d779-4b41-87e1-674b6cd98cb5 None admin] PATH_INFO: `/v3/users/9d54392a601648ee99905a3f024dd0be` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:36.074684 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e97fc1-d779-4b41-87e1-674b6cd98cb5 None admin] RBAC: Authorizing `identity:get_user(user_id=9d54392a601648ee99905a3f024dd0be)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:36.076542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e97fc1-d779-4b41-87e1-674b6cd98cb5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:36.078595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 99/200] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:35 2022] GET /identity/v3/users/9d54392a601648ee99905a3f024dd0be => generated 283 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:36.079743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e97fc1-d779-4b41-87e1-674b6cd98cb5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:36.124991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:36.194161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce86b4c-8014-4ab2-82ca-df515ccec452 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:36.250509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e97fc1-d779-4b41-87e1-674b6cd98cb5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:36.263828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2cff13-ca59-4a4f-8fad-17fadc66655c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:36.264331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2cff13-ca59-4a4f-8fad-17fadc66655c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:36.264614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2cff13-ca59-4a4f-8fad-17fadc66655c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:36.264883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2cff13-ca59-4a4f-8fad-17fadc66655c None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:36.267119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2cff13-ca59-4a4f-8fad-17fadc66655c None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:36.268615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2cff13-ca59-4a4f-8fad-17fadc66655c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:36.271564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 102/201] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:35 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:36.272436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2cff13-ca59-4a4f-8fad-17fadc66655c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:36.323684 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc057d4-cb2d-4b1c-b801-dbd6ad673623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:36.325075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc057d4-cb2d-4b1c-b801-dbd6ad673623 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:36.326129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc057d4-cb2d-4b1c-b801-dbd6ad673623 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:36.326639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc057d4-cb2d-4b1c-b801-dbd6ad673623 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:36.346598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:36.454897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc057d4-cb2d-4b1c-b801-dbd6ad673623 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:36.454897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc057d4-cb2d-4b1c-b801-dbd6ad673623 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:36.472591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2cff13-ca59-4a4f-8fad-17fadc66655c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:36.555111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a392c9-a35b-4978-b909-e0f77b5a375f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:36.555958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a392c9-a35b-4978-b909-e0f77b5a375f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:36.556283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a392c9-a35b-4978-b909-e0f77b5a375f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:36.556563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a392c9-a35b-4978-b909-e0f77b5a375f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:36.588432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:36.616787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 100/202] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 537 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:36.618513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc057d4-cb2d-4b1c-b801-dbd6ad673623 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:36.663470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a392c9-a35b-4978-b909-e0f77b5a375f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:36.666564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a392c9-a35b-4978-b909-e0f77b5a375f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:36.685143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:36.793025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc057d4-cb2d-4b1c-b801-dbd6ad673623 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:36.839192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 103/203] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 567 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:36.841169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a392c9-a35b-4978-b909-e0f77b5a375f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:36.844611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:36.844611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:36.844611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:36.844611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:36.844979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=3e92d0c573144f7986682c15b644dc35, user.id=9d54392a601648ee99905a3f024dd0be) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:36.844979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:36.846076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:36.860663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 101/204] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:36 2022] GET /identity/v3/role_assignments?user.id=9d54392a601648ee99905a3f024dd0be&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=38171c89968943bd92d9e9758db65615 => generated 259 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:36.861781 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:36.910328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:36.995087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53457be-e58e-45b6-bd0f-06e688a7fb99 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:37.003948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:37.044124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a392c9-a35b-4978-b909-e0f77b5a375f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:37.054150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaddad41-8682-4524-b711-6d39c9a748ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:37.054842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaddad41-8682-4524-b711-6d39c9a748ce None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:37.055125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaddad41-8682-4524-b711-6d39c9a748ce None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:37.055400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaddad41-8682-4524-b711-6d39c9a748ce None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:37.059658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaddad41-8682-4524-b711-6d39c9a748ce None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:37.061175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaddad41-8682-4524-b711-6d39c9a748ce None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:37.062975 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 102/205] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:36 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:37.071084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaddad41-8682-4524-b711-6d39c9a748ce None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:37.109751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea4b6fe-0277-48e0-a00e-6dd79323f92e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:37.110425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea4b6fe-0277-48e0-a00e-6dd79323f92e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:37.110714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea4b6fe-0277-48e0-a00e-6dd79323f92e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:37.111026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea4b6fe-0277-48e0-a00e-6dd79323f92e None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:37.113365 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea4b6fe-0277-48e0-a00e-6dd79323f92e None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:37.114973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea4b6fe-0277-48e0-a00e-6dd79323f92e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:37.117068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 104/206] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:36 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:37.125709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea4b6fe-0277-48e0-a00e-6dd79323f92e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:37.208163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaddad41-8682-4524-b711-6d39c9a748ce None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:37.267733 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d65cd3-72bd-4362-be46-f127358bd9da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:37.268364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d65cd3-72bd-4362-be46-f127358bd9da None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:37.268616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d65cd3-72bd-4362-be46-f127358bd9da None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:37.268887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d65cd3-72bd-4362-be46-f127358bd9da None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:37.269632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d65cd3-72bd-4362-be46-f127358bd9da None admin] RBAC: Adding query filter params (role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=3e92d0c573144f7986682c15b644dc35, user.id=9d54392a601648ee99905a3f024dd0be) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:37.269941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d65cd3-72bd-4362-be46-f127358bd9da None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:37.271467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d65cd3-72bd-4362-be46-f127358bd9da None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:37.285357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 103/207] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:37 2022] GET /identity/v3/role_assignments?user.id=9d54392a601648ee99905a3f024dd0be&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:37.347411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea4b6fe-0277-48e0-a00e-6dd79323f92e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:37.386296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:37.403319 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef18b10-81d0-48fa-8325-b364269be198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:37.403956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef18b10-81d0-48fa-8325-b364269be198 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:37.404236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef18b10-81d0-48fa-8325-b364269be198 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:37.404605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef18b10-81d0-48fa-8325-b364269be198 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:37.405366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef18b10-81d0-48fa-8325-b364269be198 None admin] RBAC: Adding query filter params (role.id=bcd685ff205446cdb68bba1cb37db029, scope.project.id=bad66f048c724ae380da0250ba68d56d, user.id=e65a84069e1d4cd7b0759a34dbfbe8dd) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:37.405675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef18b10-81d0-48fa-8325-b364269be198 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:37.407204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef18b10-81d0-48fa-8325-b364269be198 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:37.420563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 105/208] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:37 2022] GET /identity/v3/role_assignments?user.id=e65a84069e1d4cd7b0759a34dbfbe8dd&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=bcd685ff205446cdb68bba1cb37db029 => generated 608 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:37.506212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b386d9-3591-4784-9884-90c21de14ec0 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:37.506527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b386d9-3591-4784-9884-90c21de14ec0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:37.506806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b386d9-3591-4784-9884-90c21de14ec0 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:37.508178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 104/209] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:37.515699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1908d422-06f3-40fc-8258-63d069ead20d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:37.516218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1908d422-06f3-40fc-8258-63d069ead20d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:37.516591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1908d422-06f3-40fc-8258-63d069ead20d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:37.550761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1908d422-06f3-40fc-8258-63d069ead20d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:37.552831 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1908d422-06f3-40fc-8258-63d069ead20d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:37.660439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 106/210] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:37.797082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7692c7fb-2966-443f-beba-1187f95d9218 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:37.797838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7692c7fb-2966-443f-beba-1187f95d9218 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:37.798436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7692c7fb-2966-443f-beba-1187f95d9218 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:37.801340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 105/211] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:37.808138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81713aee-9b63-4452-9df2-7eacd78201b7 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:37.809012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81713aee-9b63-4452-9df2-7eacd78201b7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:37.809690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81713aee-9b63-4452-9df2-7eacd78201b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:37.845678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-81713aee-9b63-4452-9df2-7eacd78201b7 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:37.847298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-81713aee-9b63-4452-9df2-7eacd78201b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:37.964237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 107/212] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:38.090968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7692c7fb-2966-443f-beba-1187f95d9218 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:38.151183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:38.268152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7692c7fb-2966-443f-beba-1187f95d9218 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:38.314433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7379a3b2-4130-4c01-92b7-de901cdf86ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:38.315110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7379a3b2-4130-4c01-92b7-de901cdf86ef None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:38.315392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7379a3b2-4130-4c01-92b7-de901cdf86ef None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:38.315664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7379a3b2-4130-4c01-92b7-de901cdf86ef None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:38.328343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7379a3b2-4130-4c01-92b7-de901cdf86ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-7379a3b2-4130-4c01-92b7-de901cdf86ef None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:38.333803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:38.335953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:38.337986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 106/213] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:38.342767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81713aee-9b63-4452-9df2-7eacd78201b7 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:38.361494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7379a3b2-4130-4c01-92b7-de901cdf86ef None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:38.433662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:38.501889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:38.507205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7379a3b2-4130-4c01-92b7-de901cdf86ef None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:38.523334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81713aee-9b63-4452-9df2-7eacd78201b7 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:38.548076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee72b401-a63a-4675-abcb-f2d1fcf95106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:38.548721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee72b401-a63a-4675-abcb-f2d1fcf95106 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:38.549059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee72b401-a63a-4675-abcb-f2d1fcf95106 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:38.549331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee72b401-a63a-4675-abcb-f2d1fcf95106 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:38.550530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee72b401-a63a-4675-abcb-f2d1fcf95106 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:38.552281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee72b401-a63a-4675-abcb-f2d1fcf95106 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:38.554773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 107/214] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:38.563715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee72b401-a63a-4675-abcb-f2d1fcf95106 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:38.599589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:38.600629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:38.601171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:38.601595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:38.602913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:38.603466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:38.606349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:38.637892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 108/215] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:38.641070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:38.755913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee72b401-a63a-4675-abcb-f2d1fcf95106 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:38.801061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac6a40e-961a-4e66-9ee9-be4a690511f3 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:38.812241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:38.823583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6367a0af-f05d-4911-a80c-fc3d45b83407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:38.824586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6367a0af-f05d-4911-a80c-fc3d45b83407 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:38.825246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6367a0af-f05d-4911-a80c-fc3d45b83407 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:38.825891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6367a0af-f05d-4911-a80c-fc3d45b83407 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:38.826810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6367a0af-f05d-4911-a80c-fc3d45b83407 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:38.828534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6367a0af-f05d-4911-a80c-fc3d45b83407 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:38.846761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e26133-f0fe-4578-8801-3548f3b3b66f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:38.847687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e26133-f0fe-4578-8801-3548f3b3b66f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:38.847932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e26133-f0fe-4578-8801-3548f3b3b66f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:38.848348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e26133-f0fe-4578-8801-3548f3b3b66f None admin] PATH_INFO: `/v3/users/ddf90e9db2e64c9899cf58b396fb7dea` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:38.863526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 108/216] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:26:38 2022] POST /identity/v3/users => generated 297 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:26:38.874653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e26133-f0fe-4578-8801-3548f3b3b66f None admin] RBAC: Authorizing `identity:get_user(user_id=ddf90e9db2e64c9899cf58b396fb7dea)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:38.876479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e26133-f0fe-4578-8801-3548f3b3b66f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:38.878798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 109/217] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:38 2022] GET /identity/v3/users/ddf90e9db2e64c9899cf58b396fb7dea => generated 297 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:38.888184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6367a0af-f05d-4911-a80c-fc3d45b83407 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:38.922218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:39.037870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6367a0af-f05d-4911-a80c-fc3d45b83407 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:39.082294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d221218e-62ad-430d-8b69-a072e080d854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:39.083281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d221218e-62ad-430d-8b69-a072e080d854 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:39.083795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d221218e-62ad-430d-8b69-a072e080d854 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:39.084263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d221218e-62ad-430d-8b69-a072e080d854 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:39.185761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d221218e-62ad-430d-8b69-a072e080d854 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:39.188998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d221218e-62ad-430d-8b69-a072e080d854 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:39.224494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:39.288193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd90516-b2e3-42d4-8ef7-a2f26c06b89b None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:39.288700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd90516-b2e3-42d4-8ef7-a2f26c06b89b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:39.289174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd90516-b2e3-42d4-8ef7-a2f26c06b89b None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:39.291356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 110/218] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:39.297435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a753e8-e226-42a6-b7a2-4545b44587fd None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:39.297721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a753e8-e226-42a6-b7a2-4545b44587fd None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:39.298037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a753e8-e226-42a6-b7a2-4545b44587fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:39.324228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-b4a753e8-e226-42a6-b7a2-4545b44587fd None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:39.325609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b4a753e8-e226-42a6-b7a2-4545b44587fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:39.361225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 109/219] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:39.367062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d221218e-62ad-430d-8b69-a072e080d854 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:39.442573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 111/220] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:39.505195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d221218e-62ad-430d-8b69-a072e080d854 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:39.554672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673cbe70-db95-4ff7-8b6c-50cdc636d7a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:39.555373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673cbe70-db95-4ff7-8b6c-50cdc636d7a0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:39.555655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673cbe70-db95-4ff7-8b6c-50cdc636d7a0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:39.555974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673cbe70-db95-4ff7-8b6c-50cdc636d7a0 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:39.557313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673cbe70-db95-4ff7-8b6c-50cdc636d7a0 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:39.558963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673cbe70-db95-4ff7-8b6c-50cdc636d7a0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:39.563164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:39.564319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 110/221] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:39 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:39.568422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a753e8-e226-42a6-b7a2-4545b44587fd None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:39.681010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:39.688775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a753e8-e226-42a6-b7a2-4545b44587fd None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:39.732751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16162ac-c854-4423-8816-da7a2ea10cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:39.733430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16162ac-c854-4423-8816-da7a2ea10cd5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:39.733716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16162ac-c854-4423-8816-da7a2ea10cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:39.733990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16162ac-c854-4423-8816-da7a2ea10cd5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:39.734754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16162ac-c854-4423-8816-da7a2ea10cd5 None admin] RBAC: Adding query filter params (role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=3e92d0c573144f7986682c15b644dc35, user.id=ddf90e9db2e64c9899cf58b396fb7dea) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:39.735077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16162ac-c854-4423-8816-da7a2ea10cd5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:39.736549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16162ac-c854-4423-8816-da7a2ea10cd5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:39.743617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 112/222] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:39 2022] GET /identity/v3/role_assignments?user.id=ddf90e9db2e64c9899cf58b396fb7dea&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=0eb5172435534daf81351f9eb5281719 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:39.810681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673cbe70-db95-4ff7-8b6c-50cdc636d7a0 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:40.005929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673cbe70-db95-4ff7-8b6c-50cdc636d7a0 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:40.070436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2fa261-a00a-4d50-a30d-b1f80b6f1805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:40.071541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2fa261-a00a-4d50-a30d-b1f80b6f1805 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:40.072531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2fa261-a00a-4d50-a30d-b1f80b6f1805 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:40.072991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2fa261-a00a-4d50-a30d-b1f80b6f1805 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:40.080094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e9a013-3b7f-4560-9016-5253a46a0b8e None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:40.080403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e9a013-3b7f-4560-9016-5253a46a0b8e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:40.080639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e9a013-3b7f-4560-9016-5253a46a0b8e None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:40.082263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 113/223] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:40.088222 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fdb134-ba00-49a7-a762-9c42fe58e9d5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:40.088495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fdb134-ba00-49a7-a762-9c42fe58e9d5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:40.088769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fdb134-ba00-49a7-a762-9c42fe58e9d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:40.096035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2fa261-a00a-4d50-a30d-b1f80b6f1805 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2f2fa261-a00a-4d50-a30d-b1f80b6f1805 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:40.101430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:40.102504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:40.103384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:40.106662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 111/224] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:26:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:40.110814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-96fdb134-ba00-49a7-a762-9c42fe58e9d5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:40.112615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-96fdb134-ba00-49a7-a762-9c42fe58e9d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:40.116253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2fa261-a00a-4d50-a30d-b1f80b6f1805 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:40.224954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 114/225] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:40.252117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:40.304509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2fa261-a00a-4d50-a30d-b1f80b6f1805 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:40.350277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:40.350996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:40.351277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:40.351551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:40.352236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:40.352539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:40.354451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:40.368190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 112/226] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:40.370205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fdb134-ba00-49a7-a762-9c42fe58e9d5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:40.490658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:40.503306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fdb134-ba00-49a7-a762-9c42fe58e9d5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:40.560122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74cc55-77ed-4160-a971-afc4a452b319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:40.560711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74cc55-77ed-4160-a971-afc4a452b319 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:40.560996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74cc55-77ed-4160-a971-afc4a452b319 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:40.561272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74cc55-77ed-4160-a971-afc4a452b319 None admin] PATH_INFO: `/v3/users/9d54392a601648ee99905a3f024dd0be` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:40.562454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74cc55-77ed-4160-a971-afc4a452b319 None admin] RBAC: Authorizing `identity:get_user(user_id=9d54392a601648ee99905a3f024dd0be)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:40.563959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74cc55-77ed-4160-a971-afc4a452b319 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:40.565740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 115/227] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:40 2022] GET /identity/v3/users/9d54392a601648ee99905a3f024dd0be => generated 283 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:40.574724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:40.665769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74cc55-77ed-4160-a971-afc4a452b319 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:40.764979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e9c962-04d2-4ccc-81b0-54d3f5e2dfa2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:40.784058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:40.834443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecc32f2-a7e1-4323-967d-d28b767dc82a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:40.835503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecc32f2-a7e1-4323-967d-d28b767dc82a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:40.835932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecc32f2-a7e1-4323-967d-d28b767dc82a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:40.836561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecc32f2-a7e1-4323-967d-d28b767dc82a None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:40.839133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecc32f2-a7e1-4323-967d-d28b767dc82a None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:40.840955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecc32f2-a7e1-4323-967d-d28b767dc82a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:40.843374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 113/228] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:40 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:40.856500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecc32f2-a7e1-4323-967d-d28b767dc82a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:40.895383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74cc55-77ed-4160-a971-afc4a452b319 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:40.935395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558ce40-fddf-452e-8e9c-ae74eaf363af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:40.936243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558ce40-fddf-452e-8e9c-ae74eaf363af None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:40.936638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558ce40-fddf-452e-8e9c-ae74eaf363af None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:40.936986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558ce40-fddf-452e-8e9c-ae74eaf363af None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:40.949850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2558ce40-fddf-452e-8e9c-ae74eaf363af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-2558ce40-fddf-452e-8e9c-ae74eaf363af None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:40.961671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:40.963614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:40.964415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 116/229] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:26:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:40.970763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558ce40-fddf-452e-8e9c-ae74eaf363af None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:41.007583 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:41.119238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecc32f2-a7e1-4323-967d-d28b767dc82a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:41.211464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d906b8a-0aa2-4eaa-a2bf-fefd7f08ec2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:41.214121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d906b8a-0aa2-4eaa-a2bf-fefd7f08ec2a None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:41.215013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558ce40-fddf-452e-8e9c-ae74eaf363af None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:41.215548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d906b8a-0aa2-4eaa-a2bf-fefd7f08ec2a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:41.215972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d906b8a-0aa2-4eaa-a2bf-fefd7f08ec2a None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35/users/9d54392a601648ee99905a3f024dd0be/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:41.220060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d906b8a-0aa2-4eaa-a2bf-fefd7f08ec2a None admin] RBAC: Authorizing `identity:create_grant(project_id=3e92d0c573144f7986682c15b644dc35, user_id=9d54392a601648ee99905a3f024dd0be, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:41.222594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d906b8a-0aa2-4eaa-a2bf-fefd7f08ec2a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:41.249525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 114/230] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:40 2022] PUT /identity/v3/projects/3e92d0c573144f7986682c15b644dc35/users/9d54392a601648ee99905a3f024dd0be/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:41.288461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:41.289766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:41.290491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:41.291512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:41.293000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:41.293781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:41.296946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:41.327125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 117/231] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:26:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 360 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:41.332939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d906b8a-0aa2-4eaa-a2bf-fefd7f08ec2a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:41.441593 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:41.538226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d906b8a-0aa2-4eaa-a2bf-fefd7f08ec2a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:41.601406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96230c98-c312-44e0-9fb8-ae75c9bb275d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:41.602197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96230c98-c312-44e0-9fb8-ae75c9bb275d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:41.602491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96230c98-c312-44e0-9fb8-ae75c9bb275d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:41.602896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96230c98-c312-44e0-9fb8-ae75c9bb275d None admin] PATH_INFO: `/v3/users/9d54392a601648ee99905a3f024dd0be` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:41.604417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96230c98-c312-44e0-9fb8-ae75c9bb275d None admin] RBAC: Authorizing `identity:get_user(user_id=9d54392a601648ee99905a3f024dd0be)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:41.607262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96230c98-c312-44e0-9fb8-ae75c9bb275d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:41.610516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 115/232] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:41 2022] GET /identity/v3/users/9d54392a601648ee99905a3f024dd0be => generated 283 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:41.619414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:41.736167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:41.761327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827077b0-91ec-41a2-b04f-81e85755d4ab None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:41.799385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e36333-6964-4e94-93a2-96e5c8f9be90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:41.803595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e36333-6964-4e94-93a2-96e5c8f9be90 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:41.803928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e36333-6964-4e94-93a2-96e5c8f9be90 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:41.804167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e36333-6964-4e94-93a2-96e5c8f9be90 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:41.805473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e36333-6964-4e94-93a2-96e5c8f9be90 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:41.807596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e36333-6964-4e94-93a2-96e5c8f9be90 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:41.809312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 118/233] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:41 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:41.816715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96230c98-c312-44e0-9fb8-ae75c9bb275d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:42.009653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96230c98-c312-44e0-9fb8-ae75c9bb275d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:42.012416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc8ec7d-8cb0-47ce-8337-3ddb2cc7d3a5 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.013365 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc8ec7d-8cb0-47ce-8337-3ddb2cc7d3a5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.014090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc8ec7d-8cb0-47ce-8337-3ddb2cc7d3a5 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.022356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:42.023061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 119/234] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:42 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:42.025923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ed986-c531-4267-8128-ac5efa65ac0f None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.029685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ed986-c531-4267-8128-ac5efa65ac0f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.030506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ed986-c531-4267-8128-ac5efa65ac0f None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.034374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 120/235] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:42 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:42.036654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313dbe3-228b-480d-984f-3468a59531fd None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.037210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313dbe3-228b-480d-984f-3468a59531fd None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.037786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313dbe3-228b-480d-984f-3468a59531fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.057059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9efc81-7c70-4261-8c53-0dede8aa4c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:42.058175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9efc81-7c70-4261-8c53-0dede8aa4c9f None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.058614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9efc81-7c70-4261-8c53-0dede8aa4c9f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.059112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9efc81-7c70-4261-8c53-0dede8aa4c9f None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35/users/9d54392a601648ee99905a3f024dd0be/roles/f6e14365f5f54e2da132ac99f92451f5` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.061937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9efc81-7c70-4261-8c53-0dede8aa4c9f None admin] RBAC: Authorizing `identity:create_grant(project_id=3e92d0c573144f7986682c15b644dc35, user_id=9d54392a601648ee99905a3f024dd0be, role_id=f6e14365f5f54e2da132ac99f92451f5)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:42.062426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-4313dbe3-228b-480d-984f-3468a59531fd None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:42.063466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9efc81-7c70-4261-8c53-0dede8aa4c9f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:42.064118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4313dbe3-228b-480d-984f-3468a59531fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:42.091489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 116/236] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:41 2022] PUT /identity/v3/projects/3e92d0c573144f7986682c15b644dc35/users/9d54392a601648ee99905a3f024dd0be/roles/f6e14365f5f54e2da132ac99f92451f5 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:42.094266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a802e-2a29-43a5-abf3-ba3c213e6e6f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.094794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a802e-2a29-43a5-abf3-ba3c213e6e6f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.095347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a802e-2a29-43a5-abf3-ba3c213e6e6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.132090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3a9a802e-2a29-43a5-abf3-ba3c213e6e6f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:42.133918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3a9a802e-2a29-43a5-abf3-ba3c213e6e6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:42.201163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 121/237] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:42.288400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 117/238] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:42.568806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4313dbe3-228b-480d-984f-3468a59531fd None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:42.687752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:42.695544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9a802e-2a29-43a5-abf3-ba3c213e6e6f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:42.722681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4313dbe3-228b-480d-984f-3468a59531fd None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:42.773788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ea393b-b903-42bf-b0f6-db91989ebf24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:42.775986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ea393b-b903-42bf-b0f6-db91989ebf24 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.776584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ea393b-b903-42bf-b0f6-db91989ebf24 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.777170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ea393b-b903-42bf-b0f6-db91989ebf24 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.779558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ea393b-b903-42bf-b0f6-db91989ebf24 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:42.781432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ea393b-b903-42bf-b0f6-db91989ebf24 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:42.788322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 122/239] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:42.790757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adec4fa-6ce8-4e58-866f-c4faae5c6712 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.791582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adec4fa-6ce8-4e58-866f-c4faae5c6712 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.792100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adec4fa-6ce8-4e58-866f-c4faae5c6712 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.793740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 123/240] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:42.795659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adec4fa-6ce8-4e58-866f-c4faae5c6712 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:42.863825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:42.870210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9a802e-2a29-43a5-abf3-ba3c213e6e6f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:42.924773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bba56c-57a8-406b-a767-a318b7e33512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:42.925610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bba56c-57a8-406b-a767-a318b7e33512 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.925993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bba56c-57a8-406b-a767-a318b7e33512 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.926366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bba56c-57a8-406b-a767-a318b7e33512 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.930016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adec4fa-6ce8-4e58-866f-c4faae5c6712 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:42.939331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bba56c-57a8-406b-a767-a318b7e33512 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-80bba56c-57a8-406b-a767-a318b7e33512 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:42.944806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:42.945931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:42.947575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:42.949518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 118/241] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:42 2022] GET /identity/v3/roles/reader => generated 84 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:26:42.956057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f0e6de-29f9-4b3e-a59d-82fe84067589 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.956444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f0e6de-29f9-4b3e-a59d-82fe84067589 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.956822 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f0e6de-29f9-4b3e-a59d-82fe84067589 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.976802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-25f0e6de-29f9-4b3e-a59d-82fe84067589 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:42.977297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aade36c1-99b6-4a1d-8747-bb8248e61c34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:42.978244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aade36c1-99b6-4a1d-8747-bb8248e61c34 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:42.978411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-25f0e6de-29f9-4b3e-a59d-82fe84067589 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:42.978847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aade36c1-99b6-4a1d-8747-bb8248e61c34 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:42.979268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aade36c1-99b6-4a1d-8747-bb8248e61c34 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:42.980090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aade36c1-99b6-4a1d-8747-bb8248e61c34 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:42.981939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aade36c1-99b6-4a1d-8747-bb8248e61c34 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:43.015055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 124/242] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:26:42 2022] POST /identity/v3/users => generated 293 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:43.016356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aade36c1-99b6-4a1d-8747-bb8248e61c34 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:43.019782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:43.111023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 119/243] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:43.215414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aade36c1-99b6-4a1d-8747-bb8248e61c34 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:43.268279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:43.269282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:43.269759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:43.270228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:43.271192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] RBAC: Adding query filter params (name=reader) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:43.271870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:43.273874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:43.286121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 125/244] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:43.293923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f0e6de-29f9-4b3e-a59d-82fe84067589 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:43.417132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:43.432571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f0e6de-29f9-4b3e-a59d-82fe84067589 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:43.471683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9990b57-b73d-45a1-a4f0-51e3691fcc61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:43.471884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:43.472330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9990b57-b73d-45a1-a4f0-51e3691fcc61 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:43.472608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9990b57-b73d-45a1-a4f0-51e3691fcc61 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:43.472880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9990b57-b73d-45a1-a4f0-51e3691fcc61 None admin] PATH_INFO: `/v3/users/b8622bebec0b49c484a2e64cf38d9dc4` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:43.499253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9990b57-b73d-45a1-a4f0-51e3691fcc61 None admin] RBAC: Authorizing `identity:get_user(user_id=b8622bebec0b49c484a2e64cf38d9dc4)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:43.500837 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9990b57-b73d-45a1-a4f0-51e3691fcc61 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:43.502979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 120/245] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:43 2022] GET /identity/v3/users/b8622bebec0b49c484a2e64cf38d9dc4 => generated 297 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:43.508784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9990b57-b73d-45a1-a4f0-51e3691fcc61 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:43.581499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:43.654788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ff27d3-0d1c-40e8-808c-f726735d25cb None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:43.655230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9990b57-b73d-45a1-a4f0-51e3691fcc61 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:43.700440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d1bc4-e0f4-4d7b-8df7-bd5a147ced18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:43.701225 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d1bc4-e0f4-4d7b-8df7-bd5a147ced18 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:43.701520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d1bc4-e0f4-4d7b-8df7-bd5a147ced18 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:43.701907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d1bc4-e0f4-4d7b-8df7-bd5a147ced18 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:43.705894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449521ab-64a6-4b33-a957-023a567208aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:43.707013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449521ab-64a6-4b33-a957-023a567208aa None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:43.707424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449521ab-64a6-4b33-a957-023a567208aa None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:43.708244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449521ab-64a6-4b33-a957-023a567208aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:43.723147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d1bc4-e0f4-4d7b-8df7-bd5a147ced18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:43.728417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5f9d1bc4-e0f4-4d7b-8df7-bd5a147ced18 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:43.730279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:43.731705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:43.732784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 126/246] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:26:43.739032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d1bc4-e0f4-4d7b-8df7-bd5a147ced18 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:43.818724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449521ab-64a6-4b33-a957-023a567208aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:43.820419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449521ab-64a6-4b33-a957-023a567208aa None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:43.904387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d1bc4-e0f4-4d7b-8df7-bd5a147ced18 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:43.936470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:43.951497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:43.952043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:43.952328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:43.952604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:43.953302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:43.953608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:43.956160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:43.966800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 121/247] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:43.968744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 127/248] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:43.974574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449521ab-64a6-4b33-a957-023a567208aa None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:43.980169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:44.040007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:44.163745 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449521ab-64a6-4b33-a957-023a567208aa None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:44.198702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28e067-2f48-44fd-954c-cec4310980d7 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:44.233666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47db113-246c-4ece-b426-26052c532ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:44.234317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47db113-246c-4ece-b426-26052c532ad1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:44.234592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47db113-246c-4ece-b426-26052c532ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:44.234891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47db113-246c-4ece-b426-26052c532ad1 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:44.236053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47db113-246c-4ece-b426-26052c532ad1 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:44.237648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47db113-246c-4ece-b426-26052c532ad1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:44.239519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 122/249] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:43 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:44.240604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f9ccb-e5dc-4f3c-a326-3faec9953c5f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:44.240881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f9ccb-e5dc-4f3c-a326-3faec9953c5f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:44.241151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f9ccb-e5dc-4f3c-a326-3faec9953c5f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:44.242507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 123/250] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:44 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:44.248853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6783c892-cf57-4ca0-aaf6-a2c0e5ae35a2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:44.249136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6783c892-cf57-4ca0-aaf6-a2c0e5ae35a2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:44.249405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6783c892-cf57-4ca0-aaf6-a2c0e5ae35a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:44.273542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-6783c892-cf57-4ca0-aaf6-a2c0e5ae35a2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:44.275710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6783c892-cf57-4ca0-aaf6-a2c0e5ae35a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:44.305479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c5baa4-9fe3-4037-a92d-af496b462134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:44.306184 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c5baa4-9fe3-4037-a92d-af496b462134 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:44.306463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c5baa4-9fe3-4037-a92d-af496b462134 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:44.306758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c5baa4-9fe3-4037-a92d-af496b462134 None admin] PATH_INFO: `/v3/users/ddf90e9db2e64c9899cf58b396fb7dea` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:44.308654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c5baa4-9fe3-4037-a92d-af496b462134 None admin] RBAC: Authorizing `identity:get_user(user_id=ddf90e9db2e64c9899cf58b396fb7dea)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:44.312273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c5baa4-9fe3-4037-a92d-af496b462134 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:44.314712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 128/251] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:43 2022] GET /identity/v3/users/ddf90e9db2e64c9899cf58b396fb7dea => generated 297 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:44.318528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c5baa4-9fe3-4037-a92d-af496b462134 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:44.416663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 124/252] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:44.417573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6783c892-cf57-4ca0-aaf6-a2c0e5ae35a2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:44.437605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:44.511227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:44.570794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c5baa4-9fe3-4037-a92d-af496b462134 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:44.631332 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6783c892-cf57-4ca0-aaf6-a2c0e5ae35a2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:44.664411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:44.665172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:44.665508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:44.666292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:44.667275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] RBAC: Adding query filter params (role.id=bcd685ff205446cdb68bba1cb37db029, scope.project.id=3e92d0c573144f7986682c15b644dc35, user.id=b8622bebec0b49c484a2e64cf38d9dc4) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:44.668009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:44.669833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:44.687508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 129/253] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:44 2022] GET /identity/v3/role_assignments?user.id=b8622bebec0b49c484a2e64cf38d9dc4&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=bcd685ff205446cdb68bba1cb37db029 => generated 259 bytes in 371 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:44.702208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190105d3-35eb-45cf-bd17-a8437d999cc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:44.702967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190105d3-35eb-45cf-bd17-a8437d999cc7 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:44.703250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190105d3-35eb-45cf-bd17-a8437d999cc7 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:44.703527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190105d3-35eb-45cf-bd17-a8437d999cc7 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:44.704772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190105d3-35eb-45cf-bd17-a8437d999cc7 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:44.706473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190105d3-35eb-45cf-bd17-a8437d999cc7 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:44.708224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 125/254] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:44 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:44.714988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:44.729411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ba44c1-31ed-4fd4-a7b8-fe4f81ca8c1e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:44.730075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ba44c1-31ed-4fd4-a7b8-fe4f81ca8c1e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:44.730548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ba44c1-31ed-4fd4-a7b8-fe4f81ca8c1e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:44.735328 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 126/255] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:44 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:44.743195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6ceba3-d754-42ac-810d-3cae8ba7facc None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:44.744005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6ceba3-d754-42ac-810d-3cae8ba7facc None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:44.746086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6ceba3-d754-42ac-810d-3cae8ba7facc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:44.796254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-6c6ceba3-d754-42ac-810d-3cae8ba7facc None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:44.798299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6c6ceba3-d754-42ac-810d-3cae8ba7facc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:44.915532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 127/256] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:44.916757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6ceba3-d754-42ac-810d-3cae8ba7facc None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:44.963592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:44.968517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf39927-65ac-45fb-a07d-775d2aed9ee5 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:45.015892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:45.029815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6ceba3-d754-42ac-810d-3cae8ba7facc None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:45.059595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1018f6a4-8025-4cc6-98d3-ac2f82746e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:45.060310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018f6a4-8025-4cc6-98d3-ac2f82746e6c None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:45.060662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018f6a4-8025-4cc6-98d3-ac2f82746e6c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:45.061002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1018f6a4-8025-4cc6-98d3-ac2f82746e6c None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35/users/ddf90e9db2e64c9899cf58b396fb7dea/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:45.067617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1018f6a4-8025-4cc6-98d3-ac2f82746e6c None admin] RBAC: Authorizing `identity:create_grant(project_id=3e92d0c573144f7986682c15b644dc35, user_id=ddf90e9db2e64c9899cf58b396fb7dea, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:45.069383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1018f6a4-8025-4cc6-98d3-ac2f82746e6c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:45.070806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3343c4f-9583-4161-84dd-375acacd06d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:45.071534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3343c4f-9583-4161-84dd-375acacd06d6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:45.071919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3343c4f-9583-4161-84dd-375acacd06d6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:45.072262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3343c4f-9583-4161-84dd-375acacd06d6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:45.087204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3343c4f-9583-4161-84dd-375acacd06d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-e3343c4f-9583-4161-84dd-375acacd06d6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:45.092701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:45.093856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:26:45.094874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:45.098790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 128/257] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:26:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:45.098949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 130/258] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:44 2022] PUT /identity/v3/projects/3e92d0c573144f7986682c15b644dc35/users/ddf90e9db2e64c9899cf58b396fb7dea/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:45.102570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3343c4f-9583-4161-84dd-375acacd06d6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:45.224735 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3343c4f-9583-4161-84dd-375acacd06d6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:45.234469 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:45.265955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:45.266962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:45.267449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:45.267954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:45.268900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:45.269430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:45.271123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:45.282433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 129/259] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:45.292835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1018f6a4-8025-4cc6-98d3-ac2f82746e6c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:45.328320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:45.478713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1018f6a4-8025-4cc6-98d3-ac2f82746e6c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:45.479471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89566880-7962-46b0-84bb-8a06817ce0c2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:45.540202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caa3085-5bb3-46d7-bd9b-cb793692cb53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:45.541894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa3085-5bb3-46d7-bd9b-cb793692cb53 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:45.542384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa3085-5bb3-46d7-bd9b-cb793692cb53 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:45.542978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa3085-5bb3-46d7-bd9b-cb793692cb53 None admin] PATH_INFO: `/v3/users/9d54392a601648ee99905a3f024dd0be` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:45.544912 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65784d9-c5a1-488f-b469-43ab17b5ae9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:45.545823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65784d9-c5a1-488f-b469-43ab17b5ae9f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:45.546086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caa3085-5bb3-46d7-bd9b-cb793692cb53 None admin] RBAC: Authorizing `identity:get_user(user_id=9d54392a601648ee99905a3f024dd0be)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:45.546258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65784d9-c5a1-488f-b469-43ab17b5ae9f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:45.546702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65784d9-c5a1-488f-b469-43ab17b5ae9f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:45.548216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caa3085-5bb3-46d7-bd9b-cb793692cb53 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:45.550432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 131/260] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:45 2022] GET /identity/v3/users/9d54392a601648ee99905a3f024dd0be => generated 283 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:45.556626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caa3085-5bb3-46d7-bd9b-cb793692cb53 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:45.575323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65784d9-c5a1-488f-b469-43ab17b5ae9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-d65784d9-c5a1-488f-b469-43ab17b5ae9f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:45.586850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:45.588323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:45.589106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:45.596194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 130/261] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:26:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:26:45.600961 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65784d9-c5a1-488f-b469-43ab17b5ae9f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:45.652896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:45.660582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:45.778376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caa3085-5bb3-46d7-bd9b-cb793692cb53 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:45.877258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65784d9-c5a1-488f-b469-43ab17b5ae9f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:45.895379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15621d41-e01c-41f8-b9cf-36a3356e4b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:45.896011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15621d41-e01c-41f8-b9cf-36a3356e4b2f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:45.896290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15621d41-e01c-41f8-b9cf-36a3356e4b2f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:45.896563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15621d41-e01c-41f8-b9cf-36a3356e4b2f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:45.967484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:45.968115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:45.968395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:45.968667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:45.969356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:45.969656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:45.971091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:45.990363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 131/262] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:26:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 390 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:45.995864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:46.136091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15621d41-e01c-41f8-b9cf-36a3356e4b2f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:46.137529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15621d41-e01c-41f8-b9cf-36a3356e4b2f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:46.248632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:46.272398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:46.288747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269bf5ef-a974-4efd-bb04-4e0753f3e731 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:46.359266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01454dc8-08f5-4c80-8598-37097866c00b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:46.361759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01454dc8-08f5-4c80-8598-37097866c00b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:46.362046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01454dc8-08f5-4c80-8598-37097866c00b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:46.362324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01454dc8-08f5-4c80-8598-37097866c00b None admin] PATH_INFO: `/v3/users/9d54392a601648ee99905a3f024dd0be` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:46.366832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01454dc8-08f5-4c80-8598-37097866c00b None admin] RBAC: Authorizing `identity:get_user(user_id=9d54392a601648ee99905a3f024dd0be)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:46.368517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01454dc8-08f5-4c80-8598-37097866c00b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:46.374379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 132/263] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:45 2022] GET /identity/v3/users/9d54392a601648ee99905a3f024dd0be => generated 283 bytes in 377 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:46.375715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f015e-34b2-454e-889a-15a5a7cd5720 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:46.375998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f015e-34b2-454e-889a-15a5a7cd5720 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:46.376270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f015e-34b2-454e-889a-15a5a7cd5720 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:46.379469 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 133/264] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:46.381811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5f015e-34b2-454e-889a-15a5a7cd5720 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:46.397886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 132/265] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 841 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:26:46.399256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a0edb-abca-4856-a23c-5ebe2a6eb13e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:46.399540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a0edb-abca-4856-a23c-5ebe2a6eb13e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:46.399812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a0edb-abca-4856-a23c-5ebe2a6eb13e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:46.438216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2c6a0edb-abca-4856-a23c-5ebe2a6eb13e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:46.439714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2c6a0edb-abca-4856-a23c-5ebe2a6eb13e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:46.566579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 133/266] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:46.567466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6a0edb-abca-4856-a23c-5ebe2a6eb13e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:46.607678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5f015e-34b2-454e-889a-15a5a7cd5720 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:46.647152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16abc62-bb07-4ae0-903f-b1a8bbec5788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:46.647767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16abc62-bb07-4ae0-903f-b1a8bbec5788 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:46.648102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16abc62-bb07-4ae0-903f-b1a8bbec5788 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:46.648318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16abc62-bb07-4ae0-903f-b1a8bbec5788 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:46.677556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:46.730662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16abc62-bb07-4ae0-903f-b1a8bbec5788 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:46.732227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16abc62-bb07-4ae0-903f-b1a8bbec5788 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:46.778556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:46.829251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6a0edb-abca-4856-a23c-5ebe2a6eb13e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:46.848719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 134/267] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:46.850022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7f962-f958-446a-9381-166292749160 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:46.850306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7f962-f958-446a-9381-166292749160 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:46.850577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7f962-f958-446a-9381-166292749160 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:46.852044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 135/268] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:46.857047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e7f962-f958-446a-9381-166292749160 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:46.895758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d467b64e-fc92-4e34-b0a4-4f0e29c60bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:46.896433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d467b64e-fc92-4e34-b0a4-4f0e29c60bc9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:46.896702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d467b64e-fc92-4e34-b0a4-4f0e29c60bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:46.897027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d467b64e-fc92-4e34-b0a4-4f0e29c60bc9 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:46.898388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d467b64e-fc92-4e34-b0a4-4f0e29c60bc9 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:46.899885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d467b64e-fc92-4e34-b0a4-4f0e29c60bc9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:46.902530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 134/269] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:46 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:46.903819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d4a775-c496-43df-ab6d-b0f20b9c4d74 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:46.904100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d4a775-c496-43df-ab6d-b0f20b9c4d74 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:46.904369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d4a775-c496-43df-ab6d-b0f20b9c4d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:46.927655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:46.943792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-b3d4a775-c496-43df-ab6d-b0f20b9c4d74 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:46.944996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b3d4a775-c496-43df-ab6d-b0f20b9c4d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:46.978927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e7f962-f958-446a-9381-166292749160 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.018214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c48b288-811d-4a81-8229-aa74eac2866d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:47.018997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 135/270] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:47.019108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c48b288-811d-4a81-8229-aa74eac2866d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:47.019214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c48b288-811d-4a81-8229-aa74eac2866d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:47.019418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c48b288-811d-4a81-8229-aa74eac2866d None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:47.019657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d4a775-c496-43df-ab6d-b0f20b9c4d74 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:47.020570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c48b288-811d-4a81-8229-aa74eac2866d None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:47.022033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c48b288-811d-4a81-8229-aa74eac2866d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:47.023676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 136/271] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:46 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:47.024429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c48b288-811d-4a81-8229-aa74eac2866d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:47.144263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:47.181096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d4a775-c496-43df-ab6d-b0f20b9c4d74 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.187378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c48b288-811d-4a81-8229-aa74eac2866d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.215006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:47.218428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:47.219435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:47.219435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:47.219638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:47.220484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=3e92d0c573144f7986682c15b644dc35, user.id=9d54392a601648ee99905a3f024dd0be) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:47.220810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:47.222329 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:47.228760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 136/272] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:47 2022] GET /identity/v3/role_assignments?user.id=9d54392a601648ee99905a3f024dd0be&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=38171c89968943bd92d9e9758db65615 => generated 608 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:47.229213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4a1f14-a42b-48c1-9ea8-0c7fb8236823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:47.229607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:47.229779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a1f14-a42b-48c1-9ea8-0c7fb8236823 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:47.230175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a1f14-a42b-48c1-9ea8-0c7fb8236823 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:47.230451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a1f14-a42b-48c1-9ea8-0c7fb8236823 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:47.250696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4a1f14-a42b-48c1-9ea8-0c7fb8236823 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-9a4a1f14-a42b-48c1-9ea8-0c7fb8236823 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:47.260694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:47.261732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:47.262520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:47.267228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 137/273] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:47.272751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4a1f14-a42b-48c1-9ea8-0c7fb8236823 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:47.358620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c748789-ea1d-4c8e-a78f-5965a4e02ba5 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.408634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:47.409248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:47.409530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:47.409804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:47.410548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] RBAC: Adding query filter params (role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=3e92d0c573144f7986682c15b644dc35, user.id=9d54392a601648ee99905a3f024dd0be) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:47.410905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:47.412512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:47.420684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 137/274] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:47 2022] GET /identity/v3/role_assignments?user.id=9d54392a601648ee99905a3f024dd0be&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:47.464335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4a1f14-a42b-48c1-9ea8-0c7fb8236823 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.527913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:47.528814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:47.529093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:47.529366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:47.530083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:47.530479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:47.532097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:47.543542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:47.547637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 138/275] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:47.551129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:47.552985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:47.607971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:47.725143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bed97c-3ee2-44d0-8d57-1109d2799291 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.737884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1125a12e-82d7-4520-9672-1e343f33d37e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.777948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de80a66-d59b-4cd2-98ec-5b1886c6cb53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:47.778882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de80a66-d59b-4cd2-98ec-5b1886c6cb53 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:47.779162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de80a66-d59b-4cd2-98ec-5b1886c6cb53 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:47.779427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de80a66-d59b-4cd2-98ec-5b1886c6cb53 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:47.780606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de80a66-d59b-4cd2-98ec-5b1886c6cb53 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:47.782244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de80a66-d59b-4cd2-98ec-5b1886c6cb53 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:47.784883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 138/276] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:47.790321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de80a66-d59b-4cd2-98ec-5b1886c6cb53 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:47.807064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235338eb-a58e-4ea4-92c5-289bcba5c1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:47.808714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235338eb-a58e-4ea4-92c5-289bcba5c1e0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:47.809822 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235338eb-a58e-4ea4-92c5-289bcba5c1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:47.810473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235338eb-a58e-4ea4-92c5-289bcba5c1e0 None admin] PATH_INFO: `/v3/users/7fc51a1834b8430780395b4d58a74ff6` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:47.842210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235338eb-a58e-4ea4-92c5-289bcba5c1e0 None admin] RBAC: Authorizing `identity:get_user(user_id=7fc51a1834b8430780395b4d58a74ff6)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:47.844146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235338eb-a58e-4ea4-92c5-289bcba5c1e0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:47.846553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 139/277] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:47 2022] GET /identity/v3/users/7fc51a1834b8430780395b4d58a74ff6 => generated 293 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:47.856109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235338eb-a58e-4ea4-92c5-289bcba5c1e0 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:47.891050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:47.928461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de80a66-d59b-4cd2-98ec-5b1886c6cb53 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.967584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b4e157-f0eb-44dc-9c09-d0c83b2ca6c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:47.968218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4e157-f0eb-44dc-9c09-d0c83b2ca6c4 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:47.968505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4e157-f0eb-44dc-9c09-d0c83b2ca6c4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:47.968786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4e157-f0eb-44dc-9c09-d0c83b2ca6c4 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:47.969461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b4e157-f0eb-44dc-9c09-d0c83b2ca6c4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:47.970985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b4e157-f0eb-44dc-9c09-d0c83b2ca6c4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:47.987890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235338eb-a58e-4ea4-92c5-289bcba5c1e0 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:47.989650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:47.995794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 139/278] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:26:47 2022] POST /identity/v3/users => generated 293 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:48.001524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eec8e6-c072-4a7f-af51-a13dbcd71425 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:48.001524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eec8e6-c072-4a7f-af51-a13dbcd71425 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:48.001524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eec8e6-c072-4a7f-af51-a13dbcd71425 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:48.001524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 140/279] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:48.004865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e050a3-1039-494a-9ce6-706ab1a7dfea None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:48.005155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e050a3-1039-494a-9ce6-706ab1a7dfea None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:48.005461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e050a3-1039-494a-9ce6-706ab1a7dfea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:48.026254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-05e050a3-1039-494a-9ce6-706ab1a7dfea None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:48.027702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-05e050a3-1039-494a-9ce6-706ab1a7dfea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:48.052429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c6bd57-c311-4b8b-a16c-8141e510f94d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:48.053059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c6bd57-c311-4b8b-a16c-8141e510f94d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:48.053341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c6bd57-c311-4b8b-a16c-8141e510f94d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:48.053612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c6bd57-c311-4b8b-a16c-8141e510f94d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:48.054371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c6bd57-c311-4b8b-a16c-8141e510f94d None admin] RBAC: Adding query filter params (role.id=0eb5172435534daf81351f9eb5281719, user.id=7fc51a1834b8430780395b4d58a74ff6, scope.system=all) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:48.054677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c6bd57-c311-4b8b-a16c-8141e510f94d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:48.056289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c6bd57-c311-4b8b-a16c-8141e510f94d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:48.065829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 140/280] 10.222.0.77 () {56 vars in 1308 bytes} [Mon Aug 29 22:26:47 2022] GET /identity/v3/role_assignments?user.id=7fc51a1834b8430780395b4d58a74ff6&scope.system=all&role.id=0eb5172435534daf81351f9eb5281719 => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:48.065829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60acd19-5d88-4964-96e5-a094e3e2fed4 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:48.065829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60acd19-5d88-4964-96e5-a094e3e2fed4 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:48.065829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60acd19-5d88-4964-96e5-a094e3e2fed4 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:48.065829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 141/281] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:48.071602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69215ae-0d2e-4dbb-9758-e4ca78075d5a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:48.072414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69215ae-0d2e-4dbb-9758-e4ca78075d5a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:48.072673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69215ae-0d2e-4dbb-9758-e4ca78075d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:48.100060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c69215ae-0d2e-4dbb-9758-e4ca78075d5a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:48.102542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c69215ae-0d2e-4dbb-9758-e4ca78075d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:48.141883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 141/282] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:48.194755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 142/283] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:26:48.424780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e050a3-1039-494a-9ce6-706ab1a7dfea None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:48.480828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69215ae-0d2e-4dbb-9758-e4ca78075d5a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:48.522194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:48.582932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e050a3-1039-494a-9ce6-706ab1a7dfea None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:48.619593 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:48.633135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69215ae-0d2e-4dbb-9758-e4ca78075d5a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:48.649975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419e82b-c585-4c6e-bb08-9931a4b25100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:48.650937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419e82b-c585-4c6e-bb08-9931a4b25100 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:48.651414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419e82b-c585-4c6e-bb08-9931a4b25100 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:48.651823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419e82b-c585-4c6e-bb08-9931a4b25100 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:48.664614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c419e82b-c585-4c6e-bb08-9931a4b25100 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-c419e82b-c585-4c6e-bb08-9931a4b25100 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:48.670449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:48.671572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:48.672510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:48.674907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 142/284] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:48 2022] GET /identity/v3/roles/reader => generated 84 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:48.680700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419e82b-c585-4c6e-bb08-9931a4b25100 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:48.689692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3b0969-533e-44e5-af3b-b1fd56bec228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:48.691407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3b0969-533e-44e5-af3b-b1fd56bec228 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:48.692051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3b0969-533e-44e5-af3b-b1fd56bec228 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:48.692670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3b0969-533e-44e5-af3b-b1fd56bec228 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:48.719477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3b0969-533e-44e5-af3b-b1fd56bec228 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-1a3b0969-533e-44e5-af3b-b1fd56bec228 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:48.724789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:48.725866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:48.726722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:48.729834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 143/285] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:48.736833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3b0969-533e-44e5-af3b-b1fd56bec228 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:48.895843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419e82b-c585-4c6e-bb08-9931a4b25100 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:48.914427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:48.938331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:48.938902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:48.939189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:48.939460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:48.940146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] RBAC: Adding query filter params (name=reader) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:48.940450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:48.942242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:48.952328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 143/286] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:48 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:48.957695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:48.999684 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3b0969-533e-44e5-af3b-b1fd56bec228 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:49.068544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:49.069165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:49.069438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:49.069703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:49.070383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:49.070683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:49.072163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:49.095595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 144/287] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 360 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:49.104305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:49.126236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:49.172084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21468069-3a39-43a8-999b-ebb1828424bc None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:49.246059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3493f5d-68ad-4052-a858-73a66daedf4b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:49.267640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f98875f-8995-4835-b251-f8cfc237b599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:49.268683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f98875f-8995-4835-b251-f8cfc237b599 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:49.269159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f98875f-8995-4835-b251-f8cfc237b599 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:49.269621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f98875f-8995-4835-b251-f8cfc237b599 None admin] PATH_INFO: `/v3/users/b8622bebec0b49c484a2e64cf38d9dc4` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:49.271033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f98875f-8995-4835-b251-f8cfc237b599 None admin] RBAC: Authorizing `identity:get_user(user_id=b8622bebec0b49c484a2e64cf38d9dc4)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:49.272690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f98875f-8995-4835-b251-f8cfc237b599 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:49.274802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 144/288] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:48 2022] GET /identity/v3/users/b8622bebec0b49c484a2e64cf38d9dc4 => generated 297 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:49.289854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62027d6-1ce3-4555-a94b-33b46b3a0ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:49.290514 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62027d6-1ce3-4555-a94b-33b46b3a0ee4 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:49.291709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62027d6-1ce3-4555-a94b-33b46b3a0ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:49.292116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62027d6-1ce3-4555-a94b-33b46b3a0ee4 None admin] PATH_INFO: `/v3/users/ddf90e9db2e64c9899cf58b396fb7dea` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:49.292683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f98875f-8995-4835-b251-f8cfc237b599 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:49.293853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62027d6-1ce3-4555-a94b-33b46b3a0ee4 None admin] RBAC: Authorizing `identity:get_user(user_id=ddf90e9db2e64c9899cf58b396fb7dea)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:49.295712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62027d6-1ce3-4555-a94b-33b46b3a0ee4 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:49.298086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 145/289] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:49 2022] GET /identity/v3/users/ddf90e9db2e64c9899cf58b396fb7dea => generated 297 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:49.303388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62027d6-1ce3-4555-a94b-33b46b3a0ee4 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:49.368638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:49.485191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f98875f-8995-4835-b251-f8cfc237b599 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:49.519182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:49.542434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62027d6-1ce3-4555-a94b-33b46b3a0ee4 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:49.574597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dcd3b-33c5-42e5-9a49-4151c2d1c649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:49.575442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dcd3b-33c5-42e5-9a49-4151c2d1c649 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:49.575825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dcd3b-33c5-42e5-9a49-4151c2d1c649 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:49.576199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dcd3b-33c5-42e5-9a49-4151c2d1c649 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:49.577578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61dcd3b-33c5-42e5-9a49-4151c2d1c649 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:49.579938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61dcd3b-33c5-42e5-9a49-4151c2d1c649 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:49.586899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 145/290] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:49 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:49.589810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dcd3b-33c5-42e5-9a49-4151c2d1c649 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:49.687858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bbfba3-9783-449c-a410-885f3395e48f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:49.688755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bbfba3-9783-449c-a410-885f3395e48f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:49.689162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bbfba3-9783-449c-a410-885f3395e48f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:49.689611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bbfba3-9783-449c-a410-885f3395e48f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:49.716383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dcd3b-33c5-42e5-9a49-4151c2d1c649 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:49.732557 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:49.760065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32464d85-ccbf-40cb-9a9f-e844d16dad20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:49.760611 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32464d85-ccbf-40cb-9a9f-e844d16dad20 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:49.763039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32464d85-ccbf-40cb-9a9f-e844d16dad20 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:49.763331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32464d85-ccbf-40cb-9a9f-e844d16dad20 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35/users/b8622bebec0b49c484a2e64cf38d9dc4/roles/bcd685ff205446cdb68bba1cb37db029` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:49.771479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32464d85-ccbf-40cb-9a9f-e844d16dad20 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e92d0c573144f7986682c15b644dc35, user_id=b8622bebec0b49c484a2e64cf38d9dc4, role_id=bcd685ff205446cdb68bba1cb37db029)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:49.772939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32464d85-ccbf-40cb-9a9f-e844d16dad20 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:49.803838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 146/291] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:26:49 2022] PUT /identity/v3/projects/3e92d0c573144f7986682c15b644dc35/users/b8622bebec0b49c484a2e64cf38d9dc4/roles/bcd685ff205446cdb68bba1cb37db029 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:49.835062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bbfba3-9783-449c-a410-885f3395e48f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:49.836728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bbfba3-9783-449c-a410-885f3395e48f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:49.952432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:49.974721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 146/292] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 672 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:49.977190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32464d85-ccbf-40cb-9a9f-e844d16dad20 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:50.099514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32464d85-ccbf-40cb-9a9f-e844d16dad20 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:50.140389 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab0d58-d0f2-4a27-87af-8391493d3911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:50.141050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab0d58-d0f2-4a27-87af-8391493d3911 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:50.141334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab0d58-d0f2-4a27-87af-8391493d3911 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:50.141611 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab0d58-d0f2-4a27-87af-8391493d3911 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:50.142815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab0d58-d0f2-4a27-87af-8391493d3911 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:50.144361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab0d58-d0f2-4a27-87af-8391493d3911 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:50.146100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 147/293] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:49 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:50.153015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bbfba3-9783-449c-a410-885f3395e48f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:50.287341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bbfba3-9783-449c-a410-885f3395e48f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:50.335503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0e520e-63af-4cfa-a968-766262b0a08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:50.336202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e520e-63af-4cfa-a968-766262b0a08e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:50.336485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e520e-63af-4cfa-a968-766262b0a08e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:50.336759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e520e-63af-4cfa-a968-766262b0a08e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:50.337576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0e520e-63af-4cfa-a968-766262b0a08e None admin] RBAC: Adding query filter params (role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=3e92d0c573144f7986682c15b644dc35, user.id=ddf90e9db2e64c9899cf58b396fb7dea) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:50.337937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0e520e-63af-4cfa-a968-766262b0a08e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:50.339469 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0e520e-63af-4cfa-a968-766262b0a08e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:50.346287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 147/294] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:50 2022] GET /identity/v3/role_assignments?user.id=ddf90e9db2e64c9899cf58b396fb7dea&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=0eb5172435534daf81351f9eb5281719 => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:50.465025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264daa40-2949-4881-997a-9ebef21eda90 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:50.465245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264daa40-2949-4881-997a-9ebef21eda90 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:50.465554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264daa40-2949-4881-997a-9ebef21eda90 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:50.467259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 148/295] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:50.475153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4994c84-cac2-4546-8fe3-f8a111a4d6ca None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:50.475459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4994c84-cac2-4546-8fe3-f8a111a4d6ca None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:50.475706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4994c84-cac2-4546-8fe3-f8a111a4d6ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:50.503131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f4994c84-cac2-4546-8fe3-f8a111a4d6ca None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:50.505280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f4994c84-cac2-4546-8fe3-f8a111a4d6ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:50.550155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef8d471-8a56-4dbd-a096-25389096ce66 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:50.551007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef8d471-8a56-4dbd-a096-25389096ce66 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:50.551289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef8d471-8a56-4dbd-a096-25389096ce66 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:50.552637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 149/296] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:50.559111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603cd3d2-04d9-425a-8081-523739331bde None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:50.559409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603cd3d2-04d9-425a-8081-523739331bde None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:50.559659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603cd3d2-04d9-425a-8081-523739331bde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:50.590973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:50.607412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-603cd3d2-04d9-425a-8081-523739331bde None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:50.609604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-603cd3d2-04d9-425a-8081-523739331bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:50.619051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 148/297] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:50.655518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:50.693958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 150/298] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:50.796868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65805c9-5141-455a-8d31-1c46988fc6ac None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:50.797269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65805c9-5141-455a-8d31-1c46988fc6ac None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:50.797410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65805c9-5141-455a-8d31-1c46988fc6ac None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:50.799118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 149/299] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:50.805100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0654beb-0b99-4e3a-ae10-84b95bbc3308 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:50.805454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0654beb-0b99-4e3a-ae10-84b95bbc3308 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:50.805709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0654beb-0b99-4e3a-ae10-84b95bbc3308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:50.825798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f0654beb-0b99-4e3a-ae10-84b95bbc3308 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:50.827046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f0654beb-0b99-4e3a-ae10-84b95bbc3308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:50.910876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 151/300] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:50.927847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65805c9-5141-455a-8d31-1c46988fc6ac None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.058676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65805c9-5141-455a-8d31-1c46988fc6ac None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:51.099054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9998fd-db9a-48fc-85f9-b670d8f461fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:51.099951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9998fd-db9a-48fc-85f9-b670d8f461fa None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:51.100227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9998fd-db9a-48fc-85f9-b670d8f461fa None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:51.100500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9998fd-db9a-48fc-85f9-b670d8f461fa None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:51.117111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9998fd-db9a-48fc-85f9-b670d8f461fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-1e9998fd-db9a-48fc-85f9-b670d8f461fa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:51.123187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:51.124814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:51.126350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:51.127538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0654beb-0b99-4e3a-ae10-84b95bbc3308 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.127655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 150/301] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:51.131784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9998fd-db9a-48fc-85f9-b670d8f461fa None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.182121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:51.250880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9998fd-db9a-48fc-85f9-b670d8f461fa None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:51.258084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0654beb-0b99-4e3a-ae10-84b95bbc3308 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:51.260388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:51.291930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:51.292590 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:51.292922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:51.293188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:51.293863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:51.294158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:51.295600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:51.304672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 151/302] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:51.305872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.311068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3034ccc8-ae8e-4b55-b0d0-5a5c657a7b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:51.311657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3034ccc8-ae8e-4b55-b0d0-5a5c657a7b7d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:51.311987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3034ccc8-ae8e-4b55-b0d0-5a5c657a7b7d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:51.312261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3034ccc8-ae8e-4b55-b0d0-5a5c657a7b7d None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:51.313444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3034ccc8-ae8e-4b55-b0d0-5a5c657a7b7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:51.314972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3034ccc8-ae8e-4b55-b0d0-5a5c657a7b7d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:51.316716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 152/303] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:51.320475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3034ccc8-ae8e-4b55-b0d0-5a5c657a7b7d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.435902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92de56d6-3494-44f5-82ea-474bb650d3f2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:51.438075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:51.448369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3034ccc8-ae8e-4b55-b0d0-5a5c657a7b7d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:51.489652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2527b5-4c9a-4685-903d-02fa10efb762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:51.490621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2527b5-4c9a-4685-903d-02fa10efb762 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:51.491073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2527b5-4c9a-4685-903d-02fa10efb762 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:51.491559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2527b5-4c9a-4685-903d-02fa10efb762 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:51.494003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1cdf70-d508-4674-88d1-384a92115d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:51.494837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1cdf70-d508-4674-88d1-384a92115d5c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:51.495367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1cdf70-d508-4674-88d1-384a92115d5c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:51.495367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1cdf70-d508-4674-88d1-384a92115d5c None admin] PATH_INFO: `/v3/users/7fc51a1834b8430780395b4d58a74ff6` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:51.496544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1cdf70-d508-4674-88d1-384a92115d5c None admin] RBAC: Authorizing `identity:get_user(user_id=7fc51a1834b8430780395b4d58a74ff6)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:51.498000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1cdf70-d508-4674-88d1-384a92115d5c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:51.499625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 153/304] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:51 2022] GET /identity/v3/users/7fc51a1834b8430780395b4d58a74ff6 => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:51.504235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1cdf70-d508-4674-88d1-384a92115d5c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.506013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2527b5-4c9a-4685-903d-02fa10efb762 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2b2527b5-4c9a-4685-903d-02fa10efb762 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:51.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:51.514957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:51.515764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 152/305] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:51.516451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2527b5-4c9a-4685-903d-02fa10efb762 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.560754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:51.635673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1cdf70-d508-4674-88d1-384a92115d5c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:51.648569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2527b5-4c9a-4685-903d-02fa10efb762 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:51.676974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafbfc2-6184-4ffc-aea7-0da2a2ea059b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:51.677915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafbfc2-6184-4ffc-aea7-0da2a2ea059b None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:51.678561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafbfc2-6184-4ffc-aea7-0da2a2ea059b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:51.678834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafbfc2-6184-4ffc-aea7-0da2a2ea059b None admin] PATH_INFO: `/v3/groups` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:51.679486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edafbfc2-6184-4ffc-aea7-0da2a2ea059b None admin] RBAC: Authorizing `identity:create_group()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:51.680941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edafbfc2-6184-4ffc-aea7-0da2a2ea059b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:51.692373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27167fa7-2fcb-4d37-8d81-b492578227df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:51.693171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27167fa7-2fcb-4d37-8d81-b492578227df None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:51.693470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27167fa7-2fcb-4d37-8d81-b492578227df None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:51.693745 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27167fa7-2fcb-4d37-8d81-b492578227df None admin] PATH_INFO: `/v3/system/users/7fc51a1834b8430780395b4d58a74ff6/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:51.695489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27167fa7-2fcb-4d37-8d81-b492578227df None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=7fc51a1834b8430780395b4d58a74ff6, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:51.696887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27167fa7-2fcb-4d37-8d81-b492578227df None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:51.706390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 154/306] 10.222.0.77 () {60 vars in 1134 bytes} [Mon Aug 29 22:26:51 2022] POST /identity/v3/groups => generated 229 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:51.707325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafbfc2-6184-4ffc-aea7-0da2a2ea059b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.708450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 153/307] 10.222.0.77 () {58 vars in 1334 bytes} [Mon Aug 29 22:26:51 2022] PUT /identity/v3/system/users/7fc51a1834b8430780395b4d58a74ff6/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:51.849360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:51.851792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafbfc2-6184-4ffc-aea7-0da2a2ea059b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:51.902038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:51.902597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:51.902920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:51.903195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:51.904146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] RBAC: Adding query filter params (name=reader) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:51.904705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:51.906122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:51.915925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 155/308] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:51.922596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27167fa7-2fcb-4d37-8d81-b492578227df None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:51.957741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:52.054683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27167fa7-2fcb-4d37-8d81-b492578227df None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:52.093042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecdff46-101f-49e3-ad80-1f2f6aaed6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:52.093694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdff46-101f-49e3-ad80-1f2f6aaed6fb None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:52.093976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdff46-101f-49e3-ad80-1f2f6aaed6fb None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:52.094253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdff46-101f-49e3-ad80-1f2f6aaed6fb None admin] PATH_INFO: `/v3/users/cc4c4123833e429c9626ab0ae8621e62` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:52.109526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecdff46-101f-49e3-ad80-1f2f6aaed6fb None admin] RBAC: Authorizing `identity:get_user(user_id=cc4c4123833e429c9626ab0ae8621e62)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:52.111048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecdff46-101f-49e3-ad80-1f2f6aaed6fb None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:52.113187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 154/309] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:51 2022] GET /identity/v3/users/cc4c4123833e429c9626ab0ae8621e62 => generated 293 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:52.119689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:52.233445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff7b04-6586-49f2-9964-ce040ee5dd5a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:52.272633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0152ac-428d-42d2-90db-fce41cc02446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:52.273718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0152ac-428d-42d2-90db-fce41cc02446 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:52.274229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0152ac-428d-42d2-90db-fce41cc02446 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:52.274707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0152ac-428d-42d2-90db-fce41cc02446 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:52.275715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0152ac-428d-42d2-90db-fce41cc02446 None admin] RBAC: Adding query filter params (role.id=bcd685ff205446cdb68bba1cb37db029, user.id=cc4c4123833e429c9626ab0ae8621e62, scope.system=all) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:52.276241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0152ac-428d-42d2-90db-fce41cc02446 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:52.278168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0152ac-428d-42d2-90db-fce41cc02446 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:52.285112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 156/310] 10.222.0.77 () {56 vars in 1308 bytes} [Mon Aug 29 22:26:52 2022] GET /identity/v3/role_assignments?user.id=cc4c4123833e429c9626ab0ae8621e62&scope.system=all&role.id=bcd685ff205446cdb68bba1cb37db029 => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:52.351622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e912eeae-f84f-4394-ba15-02287492f900 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:52.352622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e912eeae-f84f-4394-ba15-02287492f900 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:52.353110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e912eeae-f84f-4394-ba15-02287492f900 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:52.355763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 155/311] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:52.361404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87733f69-3643-4ea7-84be-6fb2478c8da0 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:52.361701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87733f69-3643-4ea7-84be-6fb2478c8da0 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:52.361957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87733f69-3643-4ea7-84be-6fb2478c8da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:52.383451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-87733f69-3643-4ea7-84be-6fb2478c8da0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:52.385689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-87733f69-3643-4ea7-84be-6fb2478c8da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:52.407345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:52.466709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 157/312] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:52.737599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e912eeae-f84f-4394-ba15-02287492f900 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:52.843811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:52.862470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e912eeae-f84f-4394-ba15-02287492f900 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:52.901750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bde6ea-ee1d-4027-9689-0ea8db62f0e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:52.902384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bde6ea-ee1d-4027-9689-0ea8db62f0e6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:52.902662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bde6ea-ee1d-4027-9689-0ea8db62f0e6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:52.902970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bde6ea-ee1d-4027-9689-0ea8db62f0e6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:52.914327 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bde6ea-ee1d-4027-9689-0ea8db62f0e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-62bde6ea-ee1d-4027-9689-0ea8db62f0e6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:52.919910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:52.920863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:52.921661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 156/313] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:52.928867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87733f69-3643-4ea7-84be-6fb2478c8da0 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:53.049277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87733f69-3643-4ea7-84be-6fb2478c8da0 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:53.085461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:53.086439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.086956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.087474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.089084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] RBAC: Adding query filter params (name=reader) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:53.089614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:53.091243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:53.100550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 158/314] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:53.107823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bde6ea-ee1d-4027-9689-0ea8db62f0e6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:53.218054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bde6ea-ee1d-4027-9689-0ea8db62f0e6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:53.254288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ceb3bc-53ef-40bc-93e5-9ec9150d105b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:53.254959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ceb3bc-53ef-40bc-93e5-9ec9150d105b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.255240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ceb3bc-53ef-40bc-93e5-9ec9150d105b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.255511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ceb3bc-53ef-40bc-93e5-9ec9150d105b None admin] PATH_INFO: `/v3/users/b8622bebec0b49c484a2e64cf38d9dc4` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.256673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ceb3bc-53ef-40bc-93e5-9ec9150d105b None admin] RBAC: Authorizing `identity:get_user(user_id=b8622bebec0b49c484a2e64cf38d9dc4)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:53.258171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ceb3bc-53ef-40bc-93e5-9ec9150d105b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:53.260352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 157/315] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:53 2022] GET /identity/v3/users/b8622bebec0b49c484a2e64cf38d9dc4 => generated 297 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:53.266387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:53.288545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:53.380904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418d838-e3f5-4ddb-8f71-9cf865d35940 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:53.420490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d40725c-ef93-403b-9497-ae98b7215bda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:53.421496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d40725c-ef93-403b-9497-ae98b7215bda None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.421972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d40725c-ef93-403b-9497-ae98b7215bda None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.422453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d40725c-ef93-403b-9497-ae98b7215bda None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.497247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d40725c-ef93-403b-9497-ae98b7215bda None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:53.499121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d40725c-ef93-403b-9497-ae98b7215bda None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:53.520991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:53.617812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 159/316] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:53.624525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ceb3bc-53ef-40bc-93e5-9ec9150d105b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:53.641285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:53.731943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ceb3bc-53ef-40bc-93e5-9ec9150d105b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:53.756321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cea131-11ad-4ffd-8fe0-a8cc1d515688 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.756643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cea131-11ad-4ffd-8fe0-a8cc1d515688 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.756864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cea131-11ad-4ffd-8fe0-a8cc1d515688 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.758505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 160/317] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:53.765327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69682228-84df-4aea-ba1e-70cde366b71c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.765790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69682228-84df-4aea-ba1e-70cde366b71c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.766157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69682228-84df-4aea-ba1e-70cde366b71c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.772774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c859cee5-90d7-42b5-bd1b-c3f6140ef001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:53.773630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c859cee5-90d7-42b5-bd1b-c3f6140ef001 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.774047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c859cee5-90d7-42b5-bd1b-c3f6140ef001 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.774513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c859cee5-90d7-42b5-bd1b-c3f6140ef001 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.775943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c859cee5-90d7-42b5-bd1b-c3f6140ef001 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:53.777653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c859cee5-90d7-42b5-bd1b-c3f6140ef001 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:53.780284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 158/318] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:53 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:53.785793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c859cee5-90d7-42b5-bd1b-c3f6140ef001 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:53.795398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-69682228-84df-4aea-ba1e-70cde366b71c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:53.796720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-69682228-84df-4aea-ba1e-70cde366b71c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:53.881058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 161/319] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:53.887271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:53.910318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d02ee7-1188-4208-a3cc-e8c08a247fff None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.910693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d02ee7-1188-4208-a3cc-e8c08a247fff None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.911196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d02ee7-1188-4208-a3cc-e8c08a247fff None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.913809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 162/320] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:53.921535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ec5ec-e81f-4320-8495-f213df041149 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.921840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ec5ec-e81f-4320-8495-f213df041149 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.922074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ec5ec-e81f-4320-8495-f213df041149 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.924753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c859cee5-90d7-42b5-bd1b-c3f6140ef001 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:53.941844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-bf1ec5ec-e81f-4320-8495-f213df041149 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:53.943184 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-bf1ec5ec-e81f-4320-8495-f213df041149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:53.973201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:53.974196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:53.974696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:53.975215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:53.976427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] RBAC: Adding query filter params (role.id=bcd685ff205446cdb68bba1cb37db029, scope.project.id=3e92d0c573144f7986682c15b644dc35, user.id=b8622bebec0b49c484a2e64cf38d9dc4) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:53.977015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:53.979373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:53.987217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 159/321] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:26:53 2022] GET /identity/v3/role_assignments?user.id=b8622bebec0b49c484a2e64cf38d9dc4&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=bcd685ff205446cdb68bba1cb37db029 => generated 608 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:53.999825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:54.030880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 163/322] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:54.193583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:54.289525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf04f44f-ef42-4883-98ea-3cd55a4c95b9 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:54.290175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf04f44f-ef42-4883-98ea-3cd55a4c95b9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:54.290671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf04f44f-ef42-4883-98ea-3cd55a4c95b9 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:54.293035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 164/323] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:26:54.296284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf04f44f-ef42-4883-98ea-3cd55a4c95b9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:54.304341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42121a3a-55db-4f4c-9073-207a8a784137 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:54.344637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bc4fba-d592-455a-9f4d-a860cc37e56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:54.345266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bc4fba-d592-455a-9f4d-a860cc37e56b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:54.345543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bc4fba-d592-455a-9f4d-a860cc37e56b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:54.345816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bc4fba-d592-455a-9f4d-a860cc37e56b None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:54.357825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bc4fba-d592-455a-9f4d-a860cc37e56b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-e5bc4fba-d592-455a-9f4d-a860cc37e56b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:54.363316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:54.364299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:54.365084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 160/324] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:54.366181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3bf7dd-7f9d-437a-86dc-6e3c5c93705a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:54.366419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3bf7dd-7f9d-437a-86dc-6e3c5c93705a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:54.366691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3bf7dd-7f9d-437a-86dc-6e3c5c93705a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:54.374424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:54.386238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3f3bf7dd-7f9d-437a-86dc-6e3c5c93705a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:54.387511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3f3bf7dd-7f9d-437a-86dc-6e3c5c93705a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:54.417062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf04f44f-ef42-4883-98ea-3cd55a4c95b9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:54.454100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8667ca52-6a56-4a12-8117-8d9b43e837db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:54.454653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8667ca52-6a56-4a12-8117-8d9b43e837db None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:54.454959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8667ca52-6a56-4a12-8117-8d9b43e837db None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:54.455228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8667ca52-6a56-4a12-8117-8d9b43e837db None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:54.456341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8667ca52-6a56-4a12-8117-8d9b43e837db None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:54.457784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8667ca52-6a56-4a12-8117-8d9b43e837db None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:54.458343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 161/325] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:54.459130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3bf7dd-7f9d-437a-86dc-6e3c5c93705a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:54.459393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 165/326] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:26:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:54.465472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8667ca52-6a56-4a12-8117-8d9b43e837db None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:54.520311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:54.585558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8667ca52-6a56-4a12-8117-8d9b43e837db None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:54.594628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3bf7dd-7f9d-437a-86dc-6e3c5c93705a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:54.635391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b3d45b-b312-490c-a273-b7da1ee4450b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:54.636185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b3d45b-b312-490c-a273-b7da1ee4450b None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:54.636510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b3d45b-b312-490c-a273-b7da1ee4450b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:54.636777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b3d45b-b312-490c-a273-b7da1ee4450b None admin] PATH_INFO: `/v3/groups` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:54.636956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:54.637420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b3d45b-b312-490c-a273-b7da1ee4450b None admin] RBAC: Authorizing `identity:create_group()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:54.637598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:54.637827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:54.638099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:54.638797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:54.638908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b3d45b-b312-490c-a273-b7da1ee4450b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:54.639104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:54.640463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:54.642155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:54.657565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 162/327] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:54.663727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:54.667344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 166/328] 10.222.0.77 () {60 vars in 1134 bytes} [Mon Aug 29 22:26:54 2022] POST /identity/v3/groups => generated 226 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:26:54.750784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b3d45b-b312-490c-a273-b7da1ee4450b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:54.808113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d051678-e73a-46f6-83e8-38c784fdaee6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:54.851523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8731579a-2c28-4fa7-bef9-52c1f0c8ddec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:54.852160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731579a-2c28-4fa7-bef9-52c1f0c8ddec None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:54.852437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731579a-2c28-4fa7-bef9-52c1f0c8ddec None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:54.852707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731579a-2c28-4fa7-bef9-52c1f0c8ddec None admin] PATH_INFO: `/v3/users/7fc51a1834b8430780395b4d58a74ff6` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:54.853915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8731579a-2c28-4fa7-bef9-52c1f0c8ddec None admin] RBAC: Authorizing `identity:get_user(user_id=7fc51a1834b8430780395b4d58a74ff6)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:54.856369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8731579a-2c28-4fa7-bef9-52c1f0c8ddec None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:54.857700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 163/329] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:54 2022] GET /identity/v3/users/7fc51a1834b8430780395b4d58a74ff6 => generated 293 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:54.863209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8731579a-2c28-4fa7-bef9-52c1f0c8ddec None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:54.895778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b3d45b-b312-490c-a273-b7da1ee4450b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:54.899944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:54.955523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:54.955847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a726127-c33b-4f84-a684-b6a50a3a5ba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:54.956504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a726127-c33b-4f84-a684-b6a50a3a5ba5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:54.956793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a726127-c33b-4f84-a684-b6a50a3a5ba5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:54.957572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a726127-c33b-4f84-a684-b6a50a3a5ba5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:54.980771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a726127-c33b-4f84-a684-b6a50a3a5ba5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-3a726127-c33b-4f84-a684-b6a50a3a5ba5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:54.991245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:54.992368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:54.993195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 167/330] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:55.000975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a726127-c33b-4f84-a684-b6a50a3a5ba5 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:55.008727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8731579a-2c28-4fa7-bef9-52c1f0c8ddec None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:55.058112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:55.059113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:55.059113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:55.059384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:55.060159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] RBAC: Adding query filter params (role.id=0eb5172435534daf81351f9eb5281719, user.id=7fc51a1834b8430780395b4d58a74ff6, scope.system=all) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:55.060466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:55.061931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:55.076669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 164/331] 10.222.0.77 () {56 vars in 1308 bytes} [Mon Aug 29 22:26:54 2022] GET /identity/v3/role_assignments?user.id=7fc51a1834b8430780395b4d58a74ff6&scope.system=all&role.id=0eb5172435534daf81351f9eb5281719 => generated 510 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:55.157829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a726127-c33b-4f84-a684-b6a50a3a5ba5 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:55.205359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:55.205996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:55.206279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:55.206552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:55.207370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] RBAC: Adding query filter params (name=reader) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:55.207676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:55.209113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:55.218910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 168/332] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:54 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:26:55.224677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:55.351673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a371776-099c-4720-bd32-d41b1662edfe None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:55.376662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:55.393440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b11c85-1f90-42f5-aa0f-1e4009a29f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:55.394095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b11c85-1f90-42f5-aa0f-1e4009a29f7d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:55.394379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b11c85-1f90-42f5-aa0f-1e4009a29f7d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:55.394651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b11c85-1f90-42f5-aa0f-1e4009a29f7d None admin] PATH_INFO: `/v3/users/cc4c4123833e429c9626ab0ae8621e62` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:55.396783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b11c85-1f90-42f5-aa0f-1e4009a29f7d None admin] RBAC: Authorizing `identity:get_user(user_id=cc4c4123833e429c9626ab0ae8621e62)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:55.398275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b11c85-1f90-42f5-aa0f-1e4009a29f7d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:55.400217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 165/333] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:26:55 2022] GET /identity/v3/users/cc4c4123833e429c9626ab0ae8621e62 => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:55.407915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:55.453201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:55.542809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3025d3e-3639-4cf5-bce1-dff2c82e2766 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:55.581343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afb06c0-9739-4ddb-8257-6afcb33f6c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:55.582006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afb06c0-9739-4ddb-8257-6afcb33f6c60 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:55.582289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afb06c0-9739-4ddb-8257-6afcb33f6c60 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:55.582563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afb06c0-9739-4ddb-8257-6afcb33f6c60 None admin] PATH_INFO: `/v3/system/users/cc4c4123833e429c9626ab0ae8621e62/roles/bcd685ff205446cdb68bba1cb37db029` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:55.584101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afb06c0-9739-4ddb-8257-6afcb33f6c60 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=cc4c4123833e429c9626ab0ae8621e62, role_id=bcd685ff205446cdb68bba1cb37db029)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:55.585502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afb06c0-9739-4ddb-8257-6afcb33f6c60 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:55.598531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 169/334] 10.222.0.77 () {58 vars in 1334 bytes} [Mon Aug 29 22:26:55 2022] PUT /identity/v3/system/users/cc4c4123833e429c9626ab0ae8621e62/roles/bcd685ff205446cdb68bba1cb37db029 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:26:56.922595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67e29c-55f5-41e5-8825-ac4b61bbc978 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:56.923618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67e29c-55f5-41e5-8825-ac4b61bbc978 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:56.924136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67e29c-55f5-41e5-8825-ac4b61bbc978 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:56.925687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 166/335] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:56.931292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6d53b8-a22c-4ba6-9d83-2bd0418d5a3f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:56.932193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6d53b8-a22c-4ba6-9d83-2bd0418d5a3f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:56.932672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6d53b8-a22c-4ba6-9d83-2bd0418d5a3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:56.956492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-ad6d53b8-a22c-4ba6-9d83-2bd0418d5a3f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:56.958083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ad6d53b8-a22c-4ba6-9d83-2bd0418d5a3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:57.038971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 170/336] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:57.575112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a67e29c-55f5-41e5-8825-ac4b61bbc978 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:57.678168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082cac56-b24f-4598-8a96-8a7ced8ba427 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:57.678456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082cac56-b24f-4598-8a96-8a7ced8ba427 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:57.678990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082cac56-b24f-4598-8a96-8a7ced8ba427 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:57.680362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 171/337] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:57.685761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c30053-a8a1-4e8a-a390-38a13f0b1bb1 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:57.686050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c30053-a8a1-4e8a-a390-38a13f0b1bb1 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:57.686329 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c30053-a8a1-4e8a-a390-38a13f0b1bb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:57.708409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a67e29c-55f5-41e5-8825-ac4b61bbc978 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:57.715539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-76c30053-a8a1-4e8a-a390-38a13f0b1bb1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:57.716740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-76c30053-a8a1-4e8a-a390-38a13f0b1bb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:57.744190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:57.764119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d0d0a9-2bb6-45d2-9e1c-70744f6db539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:57.765212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d0d0a9-2bb6-45d2-9e1c-70744f6db539 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:57.765712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d0d0a9-2bb6-45d2-9e1c-70744f6db539 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:57.766206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d0d0a9-2bb6-45d2-9e1c-70744f6db539 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:57.807389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d0d0a9-2bb6-45d2-9e1c-70744f6db539 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-42d0d0a9-2bb6-45d2-9e1c-70744f6db539 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:57.813880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:57.814870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:57.815682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:57.818267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 167/338] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:26:57.823970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c785faab-8c8a-45fe-bfa3-ff524dfd9210 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:57.823970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c785faab-8c8a-45fe-bfa3-ff524dfd9210 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:57.823970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c785faab-8c8a-45fe-bfa3-ff524dfd9210 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:57.823970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 168/339] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:57.825340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c785faab-8c8a-45fe-bfa3-ff524dfd9210 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:57.852479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 172/340] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:57.853815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c8334-b98f-4300-aaca-41ef772a2ac9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:57.854296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c8334-b98f-4300-aaca-41ef772a2ac9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:57.854590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c8334-b98f-4300-aaca-41ef772a2ac9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:57.895871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-578c8334-b98f-4300-aaca-41ef772a2ac9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:57.897078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-578c8334-b98f-4300-aaca-41ef772a2ac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:57.904664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:57.996506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c785faab-8c8a-45fe-bfa3-ff524dfd9210 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:58.005310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 173/341] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:58.044178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:58.044805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:58.045083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:58.045353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:58.046041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:58.046347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:58.047907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:58.062192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 169/342] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:58.067452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578c8334-b98f-4300-aaca-41ef772a2ac9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:58.195431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:58.217363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578c8334-b98f-4300-aaca-41ef772a2ac9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:58.261909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7627ccc0-63ce-4ec7-a182-11f1d38e0216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:58.262521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7627ccc0-63ce-4ec7-a182-11f1d38e0216 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:58.262823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7627ccc0-63ce-4ec7-a182-11f1d38e0216 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:58.263088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7627ccc0-63ce-4ec7-a182-11f1d38e0216 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:58.349261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7627ccc0-63ce-4ec7-a182-11f1d38e0216 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:58.350814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7627ccc0-63ce-4ec7-a182-11f1d38e0216 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:58.446100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:58.470075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 174/343] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:58.475638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:58.506199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7627ccc0-63ce-4ec7-a182-11f1d38e0216 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:58.674999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7627ccc0-63ce-4ec7-a182-11f1d38e0216 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:58.709466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8114473f-211c-4cae-bb33-092ff0e09901 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:58.724554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:58.739958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9610b3-58e1-404d-8f08-8296578c9b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:58.740711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9610b3-58e1-404d-8f08-8296578c9b80 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:58.740991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9610b3-58e1-404d-8f08-8296578c9b80 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:58.741355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9610b3-58e1-404d-8f08-8296578c9b80 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:58.750095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f2616e-9a5d-4420-9b17-aa53b315fd3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:58.750757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f2616e-9a5d-4420-9b17-aa53b315fd3e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:58.751043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f2616e-9a5d-4420-9b17-aa53b315fd3e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:58.751314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f2616e-9a5d-4420-9b17-aa53b315fd3e None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:58.752633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f2616e-9a5d-4420-9b17-aa53b315fd3e None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:58.754147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f2616e-9a5d-4420-9b17-aa53b315fd3e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:58.756218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 170/344] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:58 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:58.757650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb847d91-e224-429b-985c-018b5dc52a2e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:58.757982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb847d91-e224-429b-985c-018b5dc52a2e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:58.758261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb847d91-e224-429b-985c-018b5dc52a2e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:58.759517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 171/345] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:58.760318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb847d91-e224-429b-985c-018b5dc52a2e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:58.770053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9610b3-58e1-404d-8f08-8296578c9b80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-7a9610b3-58e1-404d-8f08-8296578c9b80 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:58.775382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:58.776321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:26:58.777147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:26:58.777762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 175/346] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:26:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:26:58.778774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9610b3-58e1-404d-8f08-8296578c9b80 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:58.851666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:58.911918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb847d91-e224-429b-985c-018b5dc52a2e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:58.937484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9610b3-58e1-404d-8f08-8296578c9b80 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:58.950176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164fd0e2-bb9a-495a-be0a-2c2a0a2c8ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:58.950919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164fd0e2-bb9a-495a-be0a-2c2a0a2c8ff2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:58.951378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164fd0e2-bb9a-495a-be0a-2c2a0a2c8ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:58.951791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164fd0e2-bb9a-495a-be0a-2c2a0a2c8ff2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:58.971761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164fd0e2-bb9a-495a-be0a-2c2a0a2c8ff2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:26:58.976209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157de6d2-901a-4f07-bef4-8cb8b0b39f3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:58.976852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157de6d2-901a-4f07-bef4-8cb8b0b39f3c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:58.977137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157de6d2-901a-4f07-bef4-8cb8b0b39f3c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-164fd0e2-bb9a-495a-be0a-2c2a0a2c8ff2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:26:58.977242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:26:58.978161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:26:58.979769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:26:58.980413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157de6d2-901a-4f07-bef4-8cb8b0b39f3c None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:58.980563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 172/347] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:26:58.984601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6ab8e-9ba8-463a-9801-d9de98e28152 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:58.985220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6ab8e-9ba8-463a-9801-d9de98e28152 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:58.985520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157de6d2-901a-4f07-bef4-8cb8b0b39f3c None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:58.985803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6ab8e-9ba8-463a-9801-d9de98e28152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:58.987032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157de6d2-901a-4f07-bef4-8cb8b0b39f3c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:58.990480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 176/348] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:26:58 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:58.990480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157de6d2-901a-4f07-bef4-8cb8b0b39f3c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:59.017371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a7a6ab8e-9ba8-463a-9801-d9de98e28152 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:59.018953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a7a6ab8e-9ba8-463a-9801-d9de98e28152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:59.059123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:59.101394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 173/349] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:59.102056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a6ab8e-9ba8-463a-9801-d9de98e28152 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:59.125926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157de6d2-901a-4f07-bef4-8cb8b0b39f3c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:59.129626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:59.176592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:59.178772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:59.178772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:59.178772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:59.178772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:59.178772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:59.180136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:59.194844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 177/350] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:26:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:59.196023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:59.233988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a6ab8e-9ba8-463a-9801-d9de98e28152 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:59.287479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc72722-9ae6-4d2c-9599-c79070aaacce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:59.288474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc72722-9ae6-4d2c-9599-c79070aaacce None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:59.288962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc72722-9ae6-4d2c-9599-c79070aaacce None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:59.289437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc72722-9ae6-4d2c-9599-c79070aaacce None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:59.290348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc72722-9ae6-4d2c-9599-c79070aaacce None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:59.290887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc72722-9ae6-4d2c-9599-c79070aaacce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:59.292630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc72722-9ae6-4d2c-9599-c79070aaacce None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:59.302879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 174/351] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:26:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:59.304553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c038d1-2485-46a5-b0c0-f3cadb8c1251 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:59.305050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c038d1-2485-46a5-b0c0-f3cadb8c1251 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:59.305539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c038d1-2485-46a5-b0c0-f3cadb8c1251 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:59.309589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 175/352] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:59.310624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c038d1-2485-46a5-b0c0-f3cadb8c1251 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:59.323790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa6171f-a850-487f-89f1-d335089cb8e0 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:59.363156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:59.376516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:59.377526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:59.378003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:59.378466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:59.379454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] RBAC: Adding query filter params (group.id=6d98568fe8684cffb64ab94bdcc0f55e, role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=bad66f048c724ae380da0250ba68d56d) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:26:59.379967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:59.381806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:59.389641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 178/353] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:26:59 2022] GET /identity/v3/role_assignments?group.id=6d98568fe8684cffb64ab94bdcc0f55e&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=0eb5172435534daf81351f9eb5281719 => generated 260 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:26:59.390910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:59.433871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:59.472127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c038d1-2485-46a5-b0c0-f3cadb8c1251 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:59.532244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358a62e9-642d-4dff-a133-abf61d5731b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:59.532793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358a62e9-642d-4dff-a133-abf61d5731b2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:59.533090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358a62e9-642d-4dff-a133-abf61d5731b2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:59.533387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358a62e9-642d-4dff-a133-abf61d5731b2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:59.535615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a7ffd9-daff-4946-bfa6-ef0f4febdb25 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:26:59.583993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7582456f-d071-49f7-8d1b-a1146553dae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:26:59.584862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582456f-d071-49f7-8d1b-a1146553dae2 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:59.585221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582456f-d071-49f7-8d1b-a1146553dae2 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:59.585567 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582456f-d071-49f7-8d1b-a1146553dae2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:59.646589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358a62e9-642d-4dff-a133-abf61d5731b2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:59.648299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358a62e9-642d-4dff-a133-abf61d5731b2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:59.697140 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7582456f-d071-49f7-8d1b-a1146553dae2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:26:59.699150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:59.699718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7582456f-d071-49f7-8d1b-a1146553dae2 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:26:59.747488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:26:59.819185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 176/354] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:59.824449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1f1df8-7622-466c-b2b2-822ae212cfbb None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:59.825048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1f1df8-7622-466c-b2b2-822ae212cfbb None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:59.825526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1f1df8-7622-466c-b2b2-822ae212cfbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:59.852634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3e1f1df8-7622-466c-b2b2-822ae212cfbb None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:26:59.853334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3e1f1df8-7622-466c-b2b2-822ae212cfbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:26:59.853463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 179/355] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:26:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:26:59.858419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3e65e1-9ccf-4bf1-b2ba-b8375608e968 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:26:59.858419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3e65e1-9ccf-4bf1-b2ba-b8375608e968 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:26:59.858419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3e65e1-9ccf-4bf1-b2ba-b8375608e968 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:26:59.858419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 180/356] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:26:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:26:59.858838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3e65e1-9ccf-4bf1-b2ba-b8375608e968 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:26:59.944961 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 177/357] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:26:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:26:59.945870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1f1df8-7622-466c-b2b2-822ae212cfbb None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.020491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3e65e1-9ccf-4bf1-b2ba-b8375608e968 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.022905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:00.062393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:00.070490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e39f278-790f-48b7-a246-2e85f826bed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.071228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e39f278-790f-48b7-a246-2e85f826bed3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.071543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e39f278-790f-48b7-a246-2e85f826bed3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.071917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e39f278-790f-48b7-a246-2e85f826bed3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.083533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e39f278-790f-48b7-a246-2e85f826bed3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:00.085392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1f1df8-7622-466c-b2b2-822ae212cfbb None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5e39f278-790f-48b7-a246-2e85f826bed3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:00.089021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:00.089936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:00.090763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 181/358] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:26:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:00.092308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e39f278-790f-48b7-a246-2e85f826bed3 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.135331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144372ec-03c3-4566-8a3f-003eb13e8ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.136105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144372ec-03c3-4566-8a3f-003eb13e8ded None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.138819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144372ec-03c3-4566-8a3f-003eb13e8ded None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.139422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144372ec-03c3-4566-8a3f-003eb13e8ded None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.141605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144372ec-03c3-4566-8a3f-003eb13e8ded None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:00.143401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144372ec-03c3-4566-8a3f-003eb13e8ded None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:00.145791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 178/359] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:26:59 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:00.147930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e1c26-0e4d-442e-a370-1922900f7c9f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.148437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e1c26-0e4d-442e-a370-1922900f7c9f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.148898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e1c26-0e4d-442e-a370-1922900f7c9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.175053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b93e1c26-0e4d-442e-a370-1922900f7c9f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:00.176753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b93e1c26-0e4d-442e-a370-1922900f7c9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:00.213755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e39f278-790f-48b7-a246-2e85f826bed3 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.258985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f962180-4da3-411e-a3ec-b5488cfb234a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.259586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f962180-4da3-411e-a3ec-b5488cfb234a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.259868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f962180-4da3-411e-a3ec-b5488cfb234a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.260174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f962180-4da3-411e-a3ec-b5488cfb234a None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.261422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 179/360] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:00.262484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e1c26-0e4d-442e-a370-1922900f7c9f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.262708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f962180-4da3-411e-a3ec-b5488cfb234a None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:00.266934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f962180-4da3-411e-a3ec-b5488cfb234a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:00.266934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 182/361] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:00.267091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f962180-4da3-411e-a3ec-b5488cfb234a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.344471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:00.350336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:00.414012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e1c26-0e4d-442e-a370-1922900f7c9f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.428209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f962180-4da3-411e-a3ec-b5488cfb234a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.458409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.459569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.459939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.460414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.461296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] RBAC: Adding query filter params (name=reader) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:00.461668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:00.464459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:00.474636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81ba94b-6034-44be-9d74-e229ce9e147e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.475630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81ba94b-6034-44be-9d74-e229ce9e147e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.476315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81ba94b-6034-44be-9d74-e229ce9e147e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.476800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81ba94b-6034-44be-9d74-e229ce9e147e None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.477944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 180/362] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:00.478383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81ba94b-6034-44be-9d74-e229ce9e147e None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:00.479630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.481327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81ba94b-6034-44be-9d74-e229ce9e147e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:00.485544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 183/363] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:00.486572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81ba94b-6034-44be-9d74-e229ce9e147e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.623634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81ba94b-6034-44be-9d74-e229ce9e147e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.631543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d10730-c59b-4842-ade6-01494ab9c15a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.669442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:00.674564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:00.687043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a11f597-f198-4ca7-bea0-d9f5b39197cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.687687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a11f597-f198-4ca7-bea0-d9f5b39197cd None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.687967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a11f597-f198-4ca7-bea0-d9f5b39197cd None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.688246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a11f597-f198-4ca7-bea0-d9f5b39197cd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.689967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b50469-4486-4e3a-bdab-44aabc9ab2eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.690589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b50469-4486-4e3a-bdab-44aabc9ab2eb None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.690892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b50469-4486-4e3a-bdab-44aabc9ab2eb None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.691164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b50469-4486-4e3a-bdab-44aabc9ab2eb None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.693531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b50469-4486-4e3a-bdab-44aabc9ab2eb None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:00.695008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b50469-4486-4e3a-bdab-44aabc9ab2eb None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:00.696925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 181/364] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:00.698223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b50469-4486-4e3a-bdab-44aabc9ab2eb None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.707412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a11f597-f198-4ca7-bea0-d9f5b39197cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-3a11f597-f198-4ca7-bea0-d9f5b39197cd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:00.712513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:00.714039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:00.715109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 184/365] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:00.719660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a11f597-f198-4ca7-bea0-d9f5b39197cd None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.839515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b50469-4486-4e3a-bdab-44aabc9ab2eb None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.869156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a11f597-f198-4ca7-bea0-d9f5b39197cd None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:00.908957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3af4a56-7af7-4ec4-9c5e-015881dbf778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.909959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af4a56-7af7-4ec4-9c5e-015881dbf778 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.910447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af4a56-7af7-4ec4-9c5e-015881dbf778 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.911000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af4a56-7af7-4ec4-9c5e-015881dbf778 None admin] PATH_INFO: `/v3/users/cc4c4123833e429c9626ab0ae8621e62` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.913885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3af4a56-7af7-4ec4-9c5e-015881dbf778 None admin] RBAC: Authorizing `identity:get_user(user_id=cc4c4123833e429c9626ab0ae8621e62)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:00.914674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:00.915332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:00.915610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:00.915881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:00.916652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] RBAC: Adding query filter params (group.id=6d98568fe8684cffb64ab94bdcc0f55e, role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=bad66f048c724ae380da0250ba68d56d) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:00.916754 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3af4a56-7af7-4ec4-9c5e-015881dbf778 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:00.916958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:00.919147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:00.919956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 182/366] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/users/cc4c4123833e429c9626ab0ae8621e62 => generated 293 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:00.921175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3af4a56-7af7-4ec4-9c5e-015881dbf778 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:00.931668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 185/367] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/role_assignments?group.id=6d98568fe8684cffb64ab94bdcc0f55e&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 260 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:00.932443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:01.024338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:01.031785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:01.098887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3af4a56-7af7-4ec4-9c5e-015881dbf778 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:01.100747 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a722c308-5768-4324-96e6-b1822a366228 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:01.147664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:01.148550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.148821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.149182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.149934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] RBAC: Adding query filter params (group.id=6d98568fe8684cffb64ab94bdcc0f55e, role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=3e92d0c573144f7986682c15b644dc35) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:01.150253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:01.151753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2549cd22-cd68-487b-aafb-83f9916dddf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:01.152292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:01.152845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549cd22-cd68-487b-aafb-83f9916dddf3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.153315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549cd22-cd68-487b-aafb-83f9916dddf3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.153774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549cd22-cd68-487b-aafb-83f9916dddf3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.159117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 186/368] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/role_assignments?group.id=6d98568fe8684cffb64ab94bdcc0f55e&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=0eb5172435534daf81351f9eb5281719 => generated 260 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:01.160049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:01.172522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2549cd22-cd68-487b-aafb-83f9916dddf3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2549cd22-cd68-487b-aafb-83f9916dddf3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:01.180935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:01.181885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:01.182991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:01.185967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 183/369] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:01.187621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2549cd22-cd68-487b-aafb-83f9916dddf3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:01.314929 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c180a8-0e56-41de-87f8-8837957e4d79 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:01.336187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:01.342926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2549cd22-cd68-487b-aafb-83f9916dddf3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:01.364975 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:01.374356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:01.375030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.375318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.375592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.376286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:01.376595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:01.379687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:01.405050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 187/370] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:01.406133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:01.413726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:01.414578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.414891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.415169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.415934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] RBAC: Adding query filter params (role.id=bcd685ff205446cdb68bba1cb37db029, user.id=cc4c4123833e429c9626ab0ae8621e62, scope.system=all) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:01.416248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:01.424530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:01.445210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 184/371] 10.222.0.77 () {56 vars in 1308 bytes} [Mon Aug 29 22:27:01 2022] GET /identity/v3/role_assignments?user.id=cc4c4123833e429c9626ab0ae8621e62&scope.system=all&role.id=bcd685ff205446cdb68bba1cb37db029 => generated 510 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:01.446084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:01.644866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c789790-9124-43cc-a76a-07c24a04e5e0 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:01.673696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e28459-38da-4104-a36d-20f7a338c44e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:01.689223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53910c84-7bc6-494e-8860-df3d06406f1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:01.689945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53910c84-7bc6-494e-8860-df3d06406f1f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.690234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53910c84-7bc6-494e-8860-df3d06406f1f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.690617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53910c84-7bc6-494e-8860-df3d06406f1f None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.691347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53910c84-7bc6-494e-8860-df3d06406f1f None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:01.691659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53910c84-7bc6-494e-8860-df3d06406f1f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:01.693086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53910c84-7bc6-494e-8860-df3d06406f1f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:01.704641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 188/372] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:01.706120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfa263-c1cb-4e77-97b1-4f4c96a2d2de None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.706410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfa263-c1cb-4e77-97b1-4f4c96a2d2de None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.706813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfa263-c1cb-4e77-97b1-4f4c96a2d2de None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.708305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 189/373] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:01.710328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfa263-c1cb-4e77-97b1-4f4c96a2d2de None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:01.716534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8e7ca7-777c-4df2-8708-82d992fc8e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:01.717237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e7ca7-777c-4df2-8708-82d992fc8e2b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.717525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e7ca7-777c-4df2-8708-82d992fc8e2b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.717863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e7ca7-777c-4df2-8708-82d992fc8e2b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.766864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:01.770075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:01.811991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8e7ca7-777c-4df2-8708-82d992fc8e2b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:01.813472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8e7ca7-777c-4df2-8708-82d992fc8e2b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:01.853434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfa263-c1cb-4e77-97b1-4f4c96a2d2de None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:01.898421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5e35dc-0c0b-4e0f-aa0b-dcf779f8c6dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:01.898909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5e35dc-0c0b-4e0f-aa0b-dcf779f8c6dd None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.899190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5e35dc-0c0b-4e0f-aa0b-dcf779f8c6dd None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.899458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5e35dc-0c0b-4e0f-aa0b-dcf779f8c6dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.951528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 185/374] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:01.953141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66345d26-f96a-4479-84f1-d19a3060519b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:01.953571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66345d26-f96a-4479-84f1-d19a3060519b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:01.954043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66345d26-f96a-4479-84f1-d19a3060519b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:01.974065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-66345d26-f96a-4479-84f1-d19a3060519b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:01.975669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-66345d26-f96a-4479-84f1-d19a3060519b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:01.978644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5e35dc-0c0b-4e0f-aa0b-dcf779f8c6dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:01.980435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5e35dc-0c0b-4e0f-aa0b-dcf779f8c6dd None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:02.023637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:02.062014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 186/375] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:02.063598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66345d26-f96a-4479-84f1-d19a3060519b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:02.073547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:02.090409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 190/376] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:02.095876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5e35dc-0c0b-4e0f-aa0b-dcf779f8c6dd None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:02.198666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66345d26-f96a-4479-84f1-d19a3060519b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:02.265597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51770646-426c-48af-b5ef-2b957ef21736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:02.266526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51770646-426c-48af-b5ef-2b957ef21736 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:02.266994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51770646-426c-48af-b5ef-2b957ef21736 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:02.267500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51770646-426c-48af-b5ef-2b957ef21736 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:02.268811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51770646-426c-48af-b5ef-2b957ef21736 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:02.270430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51770646-426c-48af-b5ef-2b957ef21736 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:02.272679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 187/377] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:02 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:02.281064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51770646-426c-48af-b5ef-2b957ef21736 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:02.286467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5e35dc-0c0b-4e0f-aa0b-dcf779f8c6dd None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:02.379775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:02.395473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426d59a-e86f-477a-84d4-33651445e546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:02.396139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426d59a-e86f-477a-84d4-33651445e546 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:02.396424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426d59a-e86f-477a-84d4-33651445e546 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:02.396700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426d59a-e86f-477a-84d4-33651445e546 None admin] PATH_INFO: `/v3/projects/fabab14479bd4d4c870f8f76efa1b499` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:02.397918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2426d59a-e86f-477a-84d4-33651445e546 None admin] RBAC: Authorizing `identity:get_project(project_id=fabab14479bd4d4c870f8f76efa1b499)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:02.399436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2426d59a-e86f-477a-84d4-33651445e546 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:02.401517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 191/378] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:02 2022] GET /identity/v3/projects/fabab14479bd4d4c870f8f76efa1b499 => generated 344 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:02.407316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426d59a-e86f-477a-84d4-33651445e546 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:02.422174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51770646-426c-48af-b5ef-2b957ef21736 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:02.424269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:02.499856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069bab90-0cd0-4cd4-bf7f-c814f7de8e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:02.501327 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069bab90-0cd0-4cd4-bf7f-c814f7de8e4f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:02.502049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069bab90-0cd0-4cd4-bf7f-c814f7de8e4f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:02.502768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069bab90-0cd0-4cd4-bf7f-c814f7de8e4f None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:02.505892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069bab90-0cd0-4cd4-bf7f-c814f7de8e4f None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:02.509445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069bab90-0cd0-4cd4-bf7f-c814f7de8e4f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:02.513074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 188/379] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:02 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:02.524070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069bab90-0cd0-4cd4-bf7f-c814f7de8e4f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:02.608231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426d59a-e86f-477a-84d4-33651445e546 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:02.652652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da809237-685d-429c-b9c5-6b9e2c69e44b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:02.653460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da809237-685d-429c-b9c5-6b9e2c69e44b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:02.653834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da809237-685d-429c-b9c5-6b9e2c69e44b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:02.654198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da809237-685d-429c-b9c5-6b9e2c69e44b None admin] PATH_INFO: `/v3/groups/15cf8888d7c44182ba2e623676fc9f7d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:02.661220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da809237-685d-429c-b9c5-6b9e2c69e44b None admin] RBAC: Authorizing `identity:get_group(group_id=15cf8888d7c44182ba2e623676fc9f7d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:02.663395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da809237-685d-429c-b9c5-6b9e2c69e44b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:02.665553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 192/380] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:02 2022] GET /identity/v3/groups/15cf8888d7c44182ba2e623676fc9f7d => generated 229 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:02.667009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da809237-685d-429c-b9c5-6b9e2c69e44b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:02.677463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069bab90-0cd0-4cd4-bf7f-c814f7de8e4f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:02.726946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:02.728096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:02.728585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:02.730059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:02.731014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] RBAC: Adding query filter params (group.id=6d98568fe8684cffb64ab94bdcc0f55e, role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=3e92d0c573144f7986682c15b644dc35) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:02.731479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:02.733141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:02.745118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 189/381] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:02 2022] GET /identity/v3/role_assignments?group.id=6d98568fe8684cffb64ab94bdcc0f55e&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 260 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:02.746550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:02.756344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:02.760940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:02.839423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da809237-685d-429c-b9c5-6b9e2c69e44b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:02.900653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db37ffff-85e8-43ae-b8b2-5d97f695906c None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:02.928008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de9596-96b3-418d-953f-dead7630b5d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:02.928233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de9596-96b3-418d-953f-dead7630b5d9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:02.928476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de9596-96b3-418d-953f-dead7630b5d9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:02.928750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de9596-96b3-418d-953f-dead7630b5d9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:02.945463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:02.946337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:02.946903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:02.947494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:02.948968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] RBAC: Adding query filter params (group.id=15cf8888d7c44182ba2e623676fc9f7d, role.id=38171c89968943bd92d9e9758db65615, scope.project.id=fabab14479bd4d4c870f8f76efa1b499) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:02.949448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:02.951240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:02.958247 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 190/382] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:02 2022] GET /identity/v3/role_assignments?group.id=15cf8888d7c44182ba2e623676fc9f7d&scope.project.id=fabab14479bd4d4c870f8f76efa1b499&role.id=38171c89968943bd92d9e9758db65615 => generated 260 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:02.966846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36de9596-96b3-418d-953f-dead7630b5d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-36de9596-96b3-418d-953f-dead7630b5d9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:02.977704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:02.979494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:02.980928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 193/383] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:02.988248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:03.100694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:03.141263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79433de3-b4bb-4645-92bf-81e0fb963d13 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:03.180476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:03.181085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:03.181427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:03.181703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:03.182985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] RBAC: Adding query filter params (name=member) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:03.183292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:03.184683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:03.196183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 191/384] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:27:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:03.201994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de9596-96b3-418d-953f-dead7630b5d9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:03.339376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de9596-96b3-418d-953f-dead7630b5d9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:03.375113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:03.381467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e94106-6b5b-41f2-bc51-475a9824d6a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:03.382213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e94106-6b5b-41f2-bc51-475a9824d6a5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:03.382495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e94106-6b5b-41f2-bc51-475a9824d6a5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:03.382838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e94106-6b5b-41f2-bc51-475a9824d6a5 None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:03.384106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e94106-6b5b-41f2-bc51-475a9824d6a5 None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:03.388374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e94106-6b5b-41f2-bc51-475a9824d6a5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:03.390524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 194/385] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:03 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:03.395898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:03.542243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b8cc0-5df3-4b6f-8add-4c22990c2049 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:03.542809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b8cc0-5df3-4b6f-8add-4c22990c2049 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:03.542962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd443d7-4b92-4243-a8a2-90f886048dd7 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:03.543096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b8cc0-5df3-4b6f-8add-4c22990c2049 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:03.546117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 195/386] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:03.552535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e85eb2-b5b7-43cf-9429-f6e7a7748324 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:03.552852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e85eb2-b5b7-43cf-9429-f6e7a7748324 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:03.553186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e85eb2-b5b7-43cf-9429-f6e7a7748324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:03.592921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-72e85eb2-b5b7-43cf-9429-f6e7a7748324 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:03.594125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-72e85eb2-b5b7-43cf-9429-f6e7a7748324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:03.615148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:03.626866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfb214-9409-47db-b3eb-c9c5a53f1173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:03.627709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfb214-9409-47db-b3eb-c9c5a53f1173 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:03.627996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfb214-9409-47db-b3eb-c9c5a53f1173 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:03.628266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfb214-9409-47db-b3eb-c9c5a53f1173 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:03.629717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfb214-9409-47db-b3eb-c9c5a53f1173 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:03.632344 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfb214-9409-47db-b3eb-c9c5a53f1173 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:03.634584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 192/387] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:03 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:03.641119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfb214-9409-47db-b3eb-c9c5a53f1173 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:03.719657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 196/388] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:03.813103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfb214-9409-47db-b3eb-c9c5a53f1173 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:03.852199 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc8411d-882b-43bd-86ba-cdfc7d76bf16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:03.852866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8411d-882b-43bd-86ba-cdfc7d76bf16 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:03.853188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8411d-882b-43bd-86ba-cdfc7d76bf16 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:03.853462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8411d-882b-43bd-86ba-cdfc7d76bf16 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d/groups/6d98568fe8684cffb64ab94bdcc0f55e/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:03.856330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc8411d-882b-43bd-86ba-cdfc7d76bf16 None admin] RBAC: Authorizing `identity:create_grant(project_id=bad66f048c724ae380da0250ba68d56d, group_id=6d98568fe8684cffb64ab94bdcc0f55e, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:03.857790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc8411d-882b-43bd-86ba-cdfc7d76bf16 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:03.875301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 193/389] 10.222.0.77 () {58 vars in 1442 bytes} [Mon Aug 29 22:27:03 2022] PUT /identity/v3/projects/bad66f048c724ae380da0250ba68d56d/groups/6d98568fe8684cffb64ab94bdcc0f55e/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:03.957892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57d1f0-8e4f-4e0f-8e26-4e9d0c43098e None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:03.958215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57d1f0-8e4f-4e0f-8e26-4e9d0c43098e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:03.958393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57d1f0-8e4f-4e0f-8e26-4e9d0c43098e None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:03.960077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 197/390] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:03.972490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a345d017-992e-448e-b6ca-940eaf49cdfa None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:03.972820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a345d017-992e-448e-b6ca-940eaf49cdfa None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:03.973042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a345d017-992e-448e-b6ca-940eaf49cdfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:03.998520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a345d017-992e-448e-b6ca-940eaf49cdfa None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:03.999864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a345d017-992e-448e-b6ca-940eaf49cdfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:04.078672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:04.092137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 194/391] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:04.206387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57d1f0-8e4f-4e0f-8e26-4e9d0c43098e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:04.388380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57d1f0-8e4f-4e0f-8e26-4e9d0c43098e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:04.427901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:04.439809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb26569-db7a-4bcd-872f-86d0b66da00f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:04.440459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb26569-db7a-4bcd-872f-86d0b66da00f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:04.440742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb26569-db7a-4bcd-872f-86d0b66da00f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:04.441017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb26569-db7a-4bcd-872f-86d0b66da00f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:04.459576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb26569-db7a-4bcd-872f-86d0b66da00f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6fb26569-db7a-4bcd-872f-86d0b66da00f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:04.464844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:04.467085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:04.469125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 198/392] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:27:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:04.475631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a345d017-992e-448e-b6ca-940eaf49cdfa None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:04.504193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb26569-db7a-4bcd-872f-86d0b66da00f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:04.667311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb26569-db7a-4bcd-872f-86d0b66da00f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:04.701933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a345d017-992e-448e-b6ca-940eaf49cdfa None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:04.728902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bdad78-2788-4125-8c29-ab7bd2ee1257 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:04.730674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bdad78-2788-4125-8c29-ab7bd2ee1257 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:04.731185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bdad78-2788-4125-8c29-ab7bd2ee1257 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:04.731852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bdad78-2788-4125-8c29-ab7bd2ee1257 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:04.741444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:04.742061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:04.742339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:04.742609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:04.746762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:04.747082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:04.748493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:04.765088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bdad78-2788-4125-8c29-ab7bd2ee1257 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-90bdad78-2788-4125-8c29-ab7bd2ee1257 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:04.767043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:04.768016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:27:04.768880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:04.774751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 195/394] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:04.774874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 199/394] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:04.777228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:04.783843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bdad78-2788-4125-8c29-ab7bd2ee1257 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:04.895166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:04.897289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:04.974092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37103644-5581-4352-a90c-c36c4535c56b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:05.020459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bdad78-2788-4125-8c29-ab7bd2ee1257 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:05.062351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:05.063885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.064411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.064815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.065938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:05.066380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:05.070670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:05.079708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54ed466-bdfe-43b4-8041-6b42d434d96d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:05.080352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54ed466-bdfe-43b4-8041-6b42d434d96d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.080881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54ed466-bdfe-43b4-8041-6b42d434d96d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.081314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54ed466-bdfe-43b4-8041-6b42d434d96d None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.086591 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54ed466-bdfe-43b4-8041-6b42d434d96d None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:05.088447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54ed466-bdfe-43b4-8041-6b42d434d96d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:05.090688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 196/395] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:04 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:05.100606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 200/396] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:27:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:05.100787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54ed466-bdfe-43b4-8041-6b42d434d96d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:05.112247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:05.262280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7ad7aa-d409-48c8-b985-3b5d91c0d832 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:05.277609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:05.309908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5c989-2fd5-4777-acc4-de49b0d6aa2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:05.310892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5c989-2fd5-4777-acc4-de49b0d6aa2a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.311382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5c989-2fd5-4777-acc4-de49b0d6aa2a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.311951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5c989-2fd5-4777-acc4-de49b0d6aa2a None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.316337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b5c989-2fd5-4777-acc4-de49b0d6aa2a None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:05.318536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b5c989-2fd5-4777-acc4-de49b0d6aa2a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:05.321035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 201/397] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:05 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:05.342195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5c989-2fd5-4777-acc4-de49b0d6aa2a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:05.443575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54ed466-bdfe-43b4-8041-6b42d434d96d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:05.496194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:05.529361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96922a61-6cd4-49a4-900a-724109a15e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:05.530486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96922a61-6cd4-49a4-900a-724109a15e9c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.531827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96922a61-6cd4-49a4-900a-724109a15e9c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.533187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96922a61-6cd4-49a4-900a-724109a15e9c None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.535795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96922a61-6cd4-49a4-900a-724109a15e9c None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:05.539988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96922a61-6cd4-49a4-900a-724109a15e9c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:05.549759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 197/398] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:05 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 444 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:05.552201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb415763-3bc3-4c9b-b71b-f4061fbe9224 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.553311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb415763-3bc3-4c9b-b71b-f4061fbe9224 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.554647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb415763-3bc3-4c9b-b71b-f4061fbe9224 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.556405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 198/399] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:05 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:05.559242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb415763-3bc3-4c9b-b71b-f4061fbe9224 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:05.582762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5c989-2fd5-4777-acc4-de49b0d6aa2a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:05.668324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665d15f-c6e7-4885-bbfd-a3fe4dd7bd8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:05.668987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665d15f-c6e7-4885-bbfd-a3fe4dd7bd8c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.669066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665d15f-c6e7-4885-bbfd-a3fe4dd7bd8c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.669409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665d15f-c6e7-4885-bbfd-a3fe4dd7bd8c None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.670578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665d15f-c6e7-4885-bbfd-a3fe4dd7bd8c None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:05.671973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665d15f-c6e7-4885-bbfd-a3fe4dd7bd8c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:05.678384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 202/400] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:05 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:05.678384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e4da00-337b-4615-bc7e-3bfb6e5b64f9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.678384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e4da00-337b-4615-bc7e-3bfb6e5b64f9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.678384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e4da00-337b-4615-bc7e-3bfb6e5b64f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.703207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a6e4da00-337b-4615-bc7e-3bfb6e5b64f9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:05.703385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb415763-3bc3-4c9b-b71b-f4061fbe9224 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:05.705418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a6e4da00-337b-4615-bc7e-3bfb6e5b64f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:05.750674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bc6709-5144-45a3-92d2-bb1fab25fdd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:05.751350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bc6709-5144-45a3-92d2-bb1fab25fdd1 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.751626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bc6709-5144-45a3-92d2-bb1fab25fdd1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.751898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bc6709-5144-45a3-92d2-bb1fab25fdd1 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d/groups/6d98568fe8684cffb64ab94bdcc0f55e/roles/f6e14365f5f54e2da132ac99f92451f5` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.754034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bc6709-5144-45a3-92d2-bb1fab25fdd1 None admin] RBAC: Authorizing `identity:create_grant(project_id=bad66f048c724ae380da0250ba68d56d, group_id=6d98568fe8684cffb64ab94bdcc0f55e, role_id=f6e14365f5f54e2da132ac99f92451f5)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:05.755500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bc6709-5144-45a3-92d2-bb1fab25fdd1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:05.758062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:05.779543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 199/401] 10.222.0.77 () {58 vars in 1442 bytes} [Mon Aug 29 22:27:05 2022] PUT /identity/v3/projects/bad66f048c724ae380da0250ba68d56d/groups/6d98568fe8684cffb64ab94bdcc0f55e/roles/f6e14365f5f54e2da132ac99f92451f5 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:05.781202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e11422-0025-48c4-ae44-fefffd73c766 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.781492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e11422-0025-48c4-ae44-fefffd73c766 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.781747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e11422-0025-48c4-ae44-fefffd73c766 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.790067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 200/402] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:05.790067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e11422-0025-48c4-ae44-fefffd73c766 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:05.803075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 203/403] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:05.804782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da53a9a7-3f61-4477-8798-cdfa17fa2ee3 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:05.805065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da53a9a7-3f61-4477-8798-cdfa17fa2ee3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:05.805338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da53a9a7-3f61-4477-8798-cdfa17fa2ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:05.854424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-da53a9a7-3f61-4477-8798-cdfa17fa2ee3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:05.857121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-da53a9a7-3f61-4477-8798-cdfa17fa2ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:05.895097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:05.989753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e11422-0025-48c4-ae44-fefffd73c766 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:05.993006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 204/404] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:06.028479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1149d81d-12e5-4974-ab63-e913606c31d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:06.029160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1149d81d-12e5-4974-ab63-e913606c31d9 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:06.029442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1149d81d-12e5-4974-ab63-e913606c31d9 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:06.030055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1149d81d-12e5-4974-ab63-e913606c31d9 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35/groups/6d98568fe8684cffb64ab94bdcc0f55e/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:06.031940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1149d81d-12e5-4974-ab63-e913606c31d9 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e92d0c573144f7986682c15b644dc35, group_id=6d98568fe8684cffb64ab94bdcc0f55e, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:06.033478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1149d81d-12e5-4974-ab63-e913606c31d9 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:06.051881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 201/405] 10.222.0.77 () {58 vars in 1442 bytes} [Mon Aug 29 22:27:05 2022] PUT /identity/v3/projects/3e92d0c573144f7986682c15b644dc35/groups/6d98568fe8684cffb64ab94bdcc0f55e/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:06.111359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da53a9a7-3f61-4477-8798-cdfa17fa2ee3 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:06.227922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da53a9a7-3f61-4477-8798-cdfa17fa2ee3 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:06.237482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:06.272330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5edec8-4a63-4b75-9588-05d73ce0f3aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:06.273328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5edec8-4a63-4b75-9588-05d73ce0f3aa None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:06.273811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5edec8-4a63-4b75-9588-05d73ce0f3aa None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:06.274303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5edec8-4a63-4b75-9588-05d73ce0f3aa None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:06.287836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5edec8-4a63-4b75-9588-05d73ce0f3aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-fe5edec8-4a63-4b75-9588-05d73ce0f3aa None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:06.294744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:06.295814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:06.296936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:06.303050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 205/406] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:27:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:06.310642 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1149d81d-12e5-4974-ab63-e913606c31d9 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:06.388296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5edec8-4a63-4b75-9588-05d73ce0f3aa None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:06.475883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1149d81d-12e5-4974-ab63-e913606c31d9 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:06.487999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:06.519394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fe5001-7db1-48f2-9baa-2e00b7bf7157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:06.520377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe5001-7db1-48f2-9baa-2e00b7bf7157 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:06.520377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe5001-7db1-48f2-9baa-2e00b7bf7157 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:06.520651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe5001-7db1-48f2-9baa-2e00b7bf7157 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:06.521213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fe5001-7db1-48f2-9baa-2e00b7bf7157 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:06.521497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fe5001-7db1-48f2-9baa-2e00b7bf7157 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:06.523300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fe5001-7db1-48f2-9baa-2e00b7bf7157 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:06.541509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 202/407] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:06.541509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1919928-9e3a-4b8b-9d54-e3c3f62a3c1f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:06.541509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1919928-9e3a-4b8b-9d54-e3c3f62a3c1f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:06.541509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1919928-9e3a-4b8b-9d54-e3c3f62a3c1f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:06.544586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 203/408] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:06.548600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1919928-9e3a-4b8b-9d54-e3c3f62a3c1f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:06.564488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5edec8-4a63-4b75-9588-05d73ce0f3aa None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:06.613054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f81308-9db4-436c-8496-141aa7ed1d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:06.614328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f81308-9db4-436c-8496-141aa7ed1d9f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:06.614869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f81308-9db4-436c-8496-141aa7ed1d9f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:06.615600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f81308-9db4-436c-8496-141aa7ed1d9f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:06.632131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f81308-9db4-436c-8496-141aa7ed1d9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-f6f81308-9db4-436c-8496-141aa7ed1d9f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:06.637952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:06.639637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:06.641168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:06.645025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 206/409] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:06.645527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0a9051-bf22-46a8-b5fe-0cf3b496b530 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:06.645807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0a9051-bf22-46a8-b5fe-0cf3b496b530 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:06.646073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0a9051-bf22-46a8-b5fe-0cf3b496b530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:06.667643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1a0a9051-bf22-46a8-b5fe-0cf3b496b530 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:06.668862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1a0a9051-bf22-46a8-b5fe-0cf3b496b530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:06.693392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:06.709331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1919928-9e3a-4b8b-9d54-e3c3f62a3c1f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:06.746580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 207/410] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:06.747562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0a9051-bf22-46a8-b5fe-0cf3b496b530 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:06.763390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78427239-f55a-4dc7-a1cf-3a1e425f64c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:06.763390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78427239-f55a-4dc7-a1cf-3a1e425f64c2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:06.763390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78427239-f55a-4dc7-a1cf-3a1e425f64c2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:06.763390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78427239-f55a-4dc7-a1cf-3a1e425f64c2 None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:06.767865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78427239-f55a-4dc7-a1cf-3a1e425f64c2 None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:06.770143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78427239-f55a-4dc7-a1cf-3a1e425f64c2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:06.771821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 204/411] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:06 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:06.778586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78427239-f55a-4dc7-a1cf-3a1e425f64c2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:06.867730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:06.963058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78427239-f55a-4dc7-a1cf-3a1e425f64c2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:06.969568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0a9051-bf22-46a8-b5fe-0cf3b496b530 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:07.017788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.018408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.018683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.019029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.019712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:07.020013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:07.021380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:07.033804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 208/412] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:07.034722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:07.048459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ee7fa6-07e4-4530-a910-9636776a8320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.049091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee7fa6-07e4-4530-a910-9636776a8320 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.049369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee7fa6-07e4-4530-a910-9636776a8320 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.049640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ee7fa6-07e4-4530-a910-9636776a8320 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.050836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ee7fa6-07e4-4530-a910-9636776a8320 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:07.052296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ee7fa6-07e4-4530-a910-9636776a8320 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:07.058691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:07.059248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 205/413] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:06 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:07.060158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ee7fa6-07e4-4530-a910-9636776a8320 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:07.169081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ee7fa6-07e4-4530-a910-9636776a8320 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:07.173668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a367990-2daa-4473-b7f0-11925e0ba11b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:07.177208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:07.212154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91491a1f-d81b-4de3-b439-b04c55ff3117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.212852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91491a1f-d81b-4de3-b439-b04c55ff3117 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.213138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91491a1f-d81b-4de3-b439-b04c55ff3117 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.213417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91491a1f-d81b-4de3-b439-b04c55ff3117 None admin] PATH_INFO: `/v3/groups/15cf8888d7c44182ba2e623676fc9f7d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.214929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91491a1f-d81b-4de3-b439-b04c55ff3117 None admin] RBAC: Authorizing `identity:get_group(group_id=15cf8888d7c44182ba2e623676fc9f7d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:07.216416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91491a1f-d81b-4de3-b439-b04c55ff3117 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:07.218699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 206/414] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:07 2022] GET /identity/v3/groups/15cf8888d7c44182ba2e623676fc9f7d => generated 229 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:07.219641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91491a1f-d81b-4de3-b439-b04c55ff3117 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:07.221292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f00077-ea60-496d-a194-44f3a6fe5eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.221944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f00077-ea60-496d-a194-44f3a6fe5eb8 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.222598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f00077-ea60-496d-a194-44f3a6fe5eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.222908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f00077-ea60-496d-a194-44f3a6fe5eb8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.237378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f00077-ea60-496d-a194-44f3a6fe5eb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-92f00077-ea60-496d-a194-44f3a6fe5eb8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:07.244095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:07.245574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:27:07.246368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:07.247750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 209/415] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:07.247750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f00077-ea60-496d-a194-44f3a6fe5eb8 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:07.345064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:07.356140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91491a1f-d81b-4de3-b439-b04c55ff3117 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:07.372061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f00077-ea60-496d-a194-44f3a6fe5eb8 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:07.404696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c340f669-d254-499a-8ec3-d9b7de37d447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.406267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c340f669-d254-499a-8ec3-d9b7de37d447 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.407050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c340f669-d254-499a-8ec3-d9b7de37d447 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.407568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c340f669-d254-499a-8ec3-d9b7de37d447 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35/groups/6d98568fe8684cffb64ab94bdcc0f55e/roles/f6e14365f5f54e2da132ac99f92451f5` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.410314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c340f669-d254-499a-8ec3-d9b7de37d447 None admin] RBAC: Authorizing `identity:create_grant(project_id=3e92d0c573144f7986682c15b644dc35, group_id=6d98568fe8684cffb64ab94bdcc0f55e, role_id=f6e14365f5f54e2da132ac99f92451f5)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:07.410796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3db578-47c3-4f37-ad20-e552daae577d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.411504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3db578-47c3-4f37-ad20-e552daae577d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.411853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3db578-47c3-4f37-ad20-e552daae577d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.412198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3db578-47c3-4f37-ad20-e552daae577d None admin] PATH_INFO: `/v3/projects/fabab14479bd4d4c870f8f76efa1b499` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.412586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c340f669-d254-499a-8ec3-d9b7de37d447 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:07.413492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3db578-47c3-4f37-ad20-e552daae577d None admin] RBAC: Authorizing `identity:get_project(project_id=fabab14479bd4d4c870f8f76efa1b499)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:07.415121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3db578-47c3-4f37-ad20-e552daae577d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:07.417800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 210/416] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:07 2022] GET /identity/v3/projects/fabab14479bd4d4c870f8f76efa1b499 => generated 344 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:07.422830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3db578-47c3-4f37-ad20-e552daae577d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:07.431473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 207/417] 10.222.0.77 () {58 vars in 1442 bytes} [Mon Aug 29 22:27:07 2022] PUT /identity/v3/projects/3e92d0c573144f7986682c15b644dc35/groups/6d98568fe8684cffb64ab94bdcc0f55e/roles/f6e14365f5f54e2da132ac99f92451f5 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:07.433127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c340f669-d254-499a-8ec3-d9b7de37d447 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:07.505436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:07.625849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3db578-47c3-4f37-ad20-e552daae577d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:07.630337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c340f669-d254-499a-8ec3-d9b7de37d447 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:07.676740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc957b08-543c-48e7-8f49-ecc8f1aa615f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.677326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc957b08-543c-48e7-8f49-ecc8f1aa615f None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.677695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc957b08-543c-48e7-8f49-ecc8f1aa615f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.678011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc957b08-543c-48e7-8f49-ecc8f1aa615f None admin] PATH_INFO: `/v3/projects/fabab14479bd4d4c870f8f76efa1b499/groups/15cf8888d7c44182ba2e623676fc9f7d/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.678256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.678902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.679185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.679460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.680136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:07.680443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:07.681826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:07.684515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc957b08-543c-48e7-8f49-ecc8f1aa615f None admin] RBAC: Authorizing `identity:create_grant(project_id=fabab14479bd4d4c870f8f76efa1b499, group_id=15cf8888d7c44182ba2e623676fc9f7d, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:07.685970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc957b08-543c-48e7-8f49-ecc8f1aa615f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:07.692738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 211/418] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:27:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:07.700959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:07.705792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 208/419] 10.222.0.77 () {58 vars in 1442 bytes} [Mon Aug 29 22:27:07 2022] PUT /identity/v3/projects/fabab14479bd4d4c870f8f76efa1b499/groups/15cf8888d7c44182ba2e623676fc9f7d/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:07.715016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:07.840700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0052fb8b-7c4c-4f1b-b76c-0aa0d95c32d5 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:07.890465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7e6352-1874-4e68-8cd4-075b038545b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:07.891446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7e6352-1874-4e68-8cd4-075b038545b6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:07.891731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7e6352-1874-4e68-8cd4-075b038545b6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:07.892004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7e6352-1874-4e68-8cd4-075b038545b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:07.987977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7e6352-1874-4e68-8cd4-075b038545b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:07.989502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7e6352-1874-4e68-8cd4-075b038545b6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:08.002353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:08.120120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 212/420] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:27:08.125921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc957b08-543c-48e7-8f49-ecc8f1aa615f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:08.216118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eea2b9-bb09-4e46-8d3a-254fb74c1085 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:08.216461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eea2b9-bb09-4e46-8d3a-254fb74c1085 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:08.216675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eea2b9-bb09-4e46-8d3a-254fb74c1085 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:08.218681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 213/421] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:27:08.230124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349dfda-0372-47d9-b33a-a32c97484a21 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:08.230455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349dfda-0372-47d9-b33a-a32c97484a21 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:08.230678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349dfda-0372-47d9-b33a-a32c97484a21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:08.252830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d349dfda-0372-47d9-b33a-a32c97484a21 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:08.254546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d349dfda-0372-47d9-b33a-a32c97484a21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:08.270658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:08.281477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc957b08-543c-48e7-8f49-ecc8f1aa615f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:08.330101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99c8f3e-d0a4-484e-8195-de1d5e378f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:08.331621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c8f3e-d0a4-484e-8195-de1d5e378f83 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:08.332646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c8f3e-d0a4-484e-8195-de1d5e378f83 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:08.333133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c8f3e-d0a4-484e-8195-de1d5e378f83 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:08.335937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99c8f3e-d0a4-484e-8195-de1d5e378f83 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:08.340382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99c8f3e-d0a4-484e-8195-de1d5e378f83 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:08.345802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 209/422] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:08 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:08.348022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa29b0-0162-412b-b44a-b335cd892fd0 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:08.348491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa29b0-0162-412b-b44a-b335cd892fd0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:08.348937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa29b0-0162-412b-b44a-b335cd892fd0 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:08.352171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 210/423] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:08.353316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afa29b0-0162-412b-b44a-b335cd892fd0 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:08.359926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 214/424] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:08.361350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35d137-7037-48b5-a5c5-40b46e34f20f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:08.361644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35d137-7037-48b5-a5c5-40b46e34f20f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:08.361924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35d137-7037-48b5-a5c5-40b46e34f20f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:08.392281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-ba35d137-7037-48b5-a5c5-40b46e34f20f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:08.395034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ba35d137-7037-48b5-a5c5-40b46e34f20f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:08.447096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:08.530143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 215/425] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:08.588083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afa29b0-0162-412b-b44a-b335cd892fd0 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:08.633944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f0766d-5251-441f-be46-3b9e6dd50c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:08.634727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f0766d-5251-441f-be46-3b9e6dd50c47 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:08.634911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f0766d-5251-441f-be46-3b9e6dd50c47 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:08.635114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f0766d-5251-441f-be46-3b9e6dd50c47 None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:08.638681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f0766d-5251-441f-be46-3b9e6dd50c47 None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:08.640710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f0766d-5251-441f-be46-3b9e6dd50c47 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:08.642464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 211/426] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:08 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:08.648050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba35d137-7037-48b5-a5c5-40b46e34f20f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:08.696973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f0766d-5251-441f-be46-3b9e6dd50c47 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:08.767030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:08.836661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba35d137-7037-48b5-a5c5-40b46e34f20f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:08.846971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f0766d-5251-441f-be46-3b9e6dd50c47 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:08.882816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:08.883345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d9d3db-c5d3-4f04-b0b0-1ae9b5aa50d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:08.883467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:08.883689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:08.883960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:08.884060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d9d3db-c5d3-4f04-b0b0-1ae9b5aa50d5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:08.884244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d9d3db-c5d3-4f04-b0b0-1ae9b5aa50d5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:08.884515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d9d3db-c5d3-4f04-b0b0-1ae9b5aa50d5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:08.884687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] RBAC: Adding query filter params (group.id=6d98568fe8684cffb64ab94bdcc0f55e, role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=bad66f048c724ae380da0250ba68d56d) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:08.885001 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:08.887420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:08.898855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 216/427] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:08 2022] GET /identity/v3/role_assignments?group.id=6d98568fe8684cffb64ab94bdcc0f55e&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=0eb5172435534daf81351f9eb5281719 => generated 611 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:08.899933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:08.903673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d9d3db-c5d3-4f04-b0b0-1ae9b5aa50d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-b2d9d3db-c5d3-4f04-b0b0-1ae9b5aa50d5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:08.910058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:08.911119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:08.911934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 212/428] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:27:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:08.925102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d9d3db-c5d3-4f04-b0b0-1ae9b5aa50d5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:08.971129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:09.086684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baadbd6b-45d3-4145-b71e-979dcdeab362 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:09.093534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:09.106147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d9d3db-c5d3-4f04-b0b0-1ae9b5aa50d5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:09.125151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddd2dd3-d1e4-4de2-afee-8f1297914c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:09.126108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddd2dd3-d1e4-4de2-afee-8f1297914c3a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.126603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddd2dd3-d1e4-4de2-afee-8f1297914c3a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.127068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddd2dd3-d1e4-4de2-afee-8f1297914c3a None admin] PATH_INFO: `/v3/roles/member` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:09.138764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddd2dd3-d1e4-4de2-afee-8f1297914c3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-fddd2dd3-d1e4-4de2-afee-8f1297914c3a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:09.144080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:09.144959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 29 22:27:09.145750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:09.146355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:09.146355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.146355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.146355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:09.147237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:09.147708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:09.148726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 217/429] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:09.149299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:09.155962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddd2dd3-d1e4-4de2-afee-8f1297914c3a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:09.159094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 213/430] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:09.164648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:09.278337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:09.291682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddd2dd3-d1e4-4de2-afee-8f1297914c3a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:09.300544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c861af67-1631-47dc-bac2-fb0eddbd2c39 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:09.339525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:09.340413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.340881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.341321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:09.341894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc0957c-6c05-484e-b4b8-63e75399ff5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:09.342367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] RBAC: Adding query filter params (name=member) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:09.342662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc0957c-6c05-484e-b4b8-63e75399ff5d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.343049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc0957c-6c05-484e-b4b8-63e75399ff5d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.343208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:09.343443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc0957c-6c05-484e-b4b8-63e75399ff5d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:09.344912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:09.357192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 218/431] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:27:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:09.362832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:09.373456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:09.431579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc0957c-6c05-484e-b4b8-63e75399ff5d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:09.433308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc0957c-6c05-484e-b4b8-63e75399ff5d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:09.517553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a624bcc2-c12d-4dd3-aed3-d726eb4cba1a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:09.560236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f6d464-0f8d-4a85-be0f-482c2c8c2734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:09.561098 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f6d464-0f8d-4a85-be0f-482c2c8c2734 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.561662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f6d464-0f8d-4a85-be0f-482c2c8c2734 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.562244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f6d464-0f8d-4a85-be0f-482c2c8c2734 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:09.580431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 214/432] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:27:09.586623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc0957c-6c05-484e-b4b8-63e75399ff5d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:09.596416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:09.658579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f6d464-0f8d-4a85-be0f-482c2c8c2734 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:09.659182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:09.662128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f6d464-0f8d-4a85-be0f-482c2c8c2734 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:09.714497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc0957c-6c05-484e-b4b8-63e75399ff5d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:09.753979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b478e97f-be6e-45d2-a463-cdc96ce284a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:09.754933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b478e97f-be6e-45d2-a463-cdc96ce284a8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.755385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b478e97f-be6e-45d2-a463-cdc96ce284a8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.755824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b478e97f-be6e-45d2-a463-cdc96ce284a8 None admin] PATH_INFO: `/v3/projects/bad66f048c724ae380da0250ba68d56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:09.757266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b478e97f-be6e-45d2-a463-cdc96ce284a8 None admin] RBAC: Authorizing `identity:get_project(project_id=bad66f048c724ae380da0250ba68d56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:09.759051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b478e97f-be6e-45d2-a463-cdc96ce284a8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:09.761723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 215/433] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:09 2022] GET /identity/v3/projects/bad66f048c724ae380da0250ba68d56d => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:09.768471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b478e97f-be6e-45d2-a463-cdc96ce284a8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:09.802580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 219/434] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:27:09.807700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f6d464-0f8d-4a85-be0f-482c2c8c2734 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:09.894897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:09.908712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b478e97f-be6e-45d2-a463-cdc96ce284a8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:09.940101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:09.952875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f6d464-0f8d-4a85-be0f-482c2c8c2734 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:09.960153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3816c5fc-c6fe-402d-ad57-4095c95df875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:09.960930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3816c5fc-c6fe-402d-ad57-4095c95df875 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.961310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3816c5fc-c6fe-402d-ad57-4095c95df875 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.961681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3816c5fc-c6fe-402d-ad57-4095c95df875 None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:09.962996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3816c5fc-c6fe-402d-ad57-4095c95df875 None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:09.965131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3816c5fc-c6fe-402d-ad57-4095c95df875 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:09.966873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 216/435] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:09 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:09.970778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc9d2c-8e06-420b-a18e-345337b62344 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.970778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc9d2c-8e06-420b-a18e-345337b62344 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.970778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc9d2c-8e06-420b-a18e-345337b62344 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:09.974119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 217/436] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:09.975548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216bb94a-f79b-4e21-8fc9-0b46b2c0a138 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:09.975932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216bb94a-f79b-4e21-8fc9-0b46b2c0a138 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:09.977043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216bb94a-f79b-4e21-8fc9-0b46b2c0a138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.002611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-216bb94a-f79b-4e21-8fc9-0b46b2c0a138 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:10.004651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-216bb94a-f79b-4e21-8fc9-0b46b2c0a138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:10.010258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fd983-fbbf-4d43-a342-5659751d255b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:10.011862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fd983-fbbf-4d43-a342-5659751d255b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.012352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fd983-fbbf-4d43-a342-5659751d255b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.012876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fd983-fbbf-4d43-a342-5659751d255b None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.014875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97fd983-fbbf-4d43-a342-5659751d255b None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:10.016589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97fd983-fbbf-4d43-a342-5659751d255b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:10.018834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 220/437] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:09 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:10.019817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fd983-fbbf-4d43-a342-5659751d255b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:10.094924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 218/438] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:10.095793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216bb94a-f79b-4e21-8fc9-0b46b2c0a138 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:10.160556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fd983-fbbf-4d43-a342-5659751d255b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:10.186972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:10.201158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215b7311-54b8-40c0-bf05-5f258f460f4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:10.201977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215b7311-54b8-40c0-bf05-5f258f460f4e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.202086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215b7311-54b8-40c0-bf05-5f258f460f4e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.202297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215b7311-54b8-40c0-bf05-5f258f460f4e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.203057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215b7311-54b8-40c0-bf05-5f258f460f4e None admin] RBAC: Adding query filter params (group.id=6d98568fe8684cffb64ab94bdcc0f55e, role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=bad66f048c724ae380da0250ba68d56d) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:10.203381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215b7311-54b8-40c0-bf05-5f258f460f4e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:10.204811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215b7311-54b8-40c0-bf05-5f258f460f4e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:10.211191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 221/439] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:10 2022] GET /identity/v3/role_assignments?group.id=6d98568fe8684cffb64ab94bdcc0f55e&scope.project.id=bad66f048c724ae380da0250ba68d56d&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 611 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:10.212188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7dd0e-dae8-4085-96cd-fee487d0d828 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.212644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7dd0e-dae8-4085-96cd-fee487d0d828 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.212749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7dd0e-dae8-4085-96cd-fee487d0d828 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.214282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 222/440] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:10.222850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216bb94a-f79b-4e21-8fc9-0b46b2c0a138 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:10.223058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93393db-4065-4e87-940a-576bcba9faf3 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.223058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93393db-4065-4e87-940a-576bcba9faf3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.223058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93393db-4065-4e87-940a-576bcba9faf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.248450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:10.254666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a93393db-4065-4e87-940a-576bcba9faf3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:10.258155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a93393db-4065-4e87-940a-576bcba9faf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:10.272114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c318931a-c690-4d10-b976-00bb1325a72f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:10.272891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318931a-c690-4d10-b976-00bb1325a72f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.273242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318931a-c690-4d10-b976-00bb1325a72f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.273584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c318931a-c690-4d10-b976-00bb1325a72f None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.275078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c318931a-c690-4d10-b976-00bb1325a72f None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:10.277600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c318931a-c690-4d10-b976-00bb1325a72f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:10.280985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 219/441] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:10 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:10.286692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c318931a-c690-4d10-b976-00bb1325a72f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:10.364511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 223/442] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:10.402380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93393db-4065-4e87-940a-576bcba9faf3 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:10.437517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c318931a-c690-4d10-b976-00bb1325a72f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:10.485457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:10.486081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.486366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.486802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.487507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] RBAC: Adding query filter params (group.id=6d98568fe8684cffb64ab94bdcc0f55e, role.id=0eb5172435534daf81351f9eb5281719, scope.project.id=3e92d0c573144f7986682c15b644dc35) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:10.487826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:10.489313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:10.496673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 220/443] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:10 2022] GET /identity/v3/role_assignments?group.id=6d98568fe8684cffb64ab94bdcc0f55e&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=0eb5172435534daf81351f9eb5281719 => generated 611 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:10.551390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93393db-4065-4e87-940a-576bcba9faf3 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:10.577257 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:10.594634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992735a-db48-4008-865d-37ce28826831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:10.595328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992735a-db48-4008-865d-37ce28826831 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.595608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992735a-db48-4008-865d-37ce28826831 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.595880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992735a-db48-4008-865d-37ce28826831 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.609387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8992735a-db48-4008-865d-37ce28826831 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-8992735a-db48-4008-865d-37ce28826831 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:10.614675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:10.615670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:10.616697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 224/444] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:27:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:10.622643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:10.649720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992735a-db48-4008-865d-37ce28826831 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:10.690263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:10.770654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14263c6f-3e36-469a-9632-bae88b8bfb64 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:10.784820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992735a-db48-4008-865d-37ce28826831 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:10.818575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:10.819209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.819630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.819732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.820413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:10.820711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:10.822100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:10.834258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 221/445] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:10.839273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cb2da0-794e-4571-a171-66964cba0757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:10.839273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cb2da0-794e-4571-a171-66964cba0757 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:10.839273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cb2da0-794e-4571-a171-66964cba0757 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:10.839273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cb2da0-794e-4571-a171-66964cba0757 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:10.841415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:10.850652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cb2da0-794e-4571-a171-66964cba0757 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-c0cb2da0-794e-4571-a171-66964cba0757 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:10.856760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:10.857664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:10.858465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 225/446] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:10.863236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cb2da0-794e-4571-a171-66964cba0757 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:10.908039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:10.962812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da186a3-e81a-47fe-ba04-9d03c5862839 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:10.977300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:10.979115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cb2da0-794e-4571-a171-66964cba0757 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:11.002430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10d90a3-c39a-4e66-9b35-b077a57234f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.003064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10d90a3-c39a-4e66-9b35-b077a57234f3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.003341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10d90a3-c39a-4e66-9b35-b077a57234f3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.003618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10d90a3-c39a-4e66-9b35-b077a57234f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.018644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.019294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.019728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.019997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.021072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:11.021521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.022938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.032688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 226/447] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:11.038243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:11.085256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10d90a3-c39a-4e66-9b35-b077a57234f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.086930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10d90a3-c39a-4e66-9b35-b077a57234f3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.141682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:11.145779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b390163-17eb-4676-894c-4fabb43a2b5d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:11.180977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85041d66-480e-425f-89f9-00e6649df94a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.181588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85041d66-480e-425f-89f9-00e6649df94a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.181873 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85041d66-480e-425f-89f9-00e6649df94a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.182146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85041d66-480e-425f-89f9-00e6649df94a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.197620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 222/448] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:11.203887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10d90a3-c39a-4e66-9b35-b077a57234f3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:11.217993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:11.253029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85041d66-480e-425f-89f9-00e6649df94a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.254487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85041d66-480e-425f-89f9-00e6649df94a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.303994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10d90a3-c39a-4e66-9b35-b077a57234f3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:11.337937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cf6ae9-cfd6-4e37-bf3e-e8ee49b54826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.338555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cf6ae9-cfd6-4e37-bf3e-e8ee49b54826 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.339408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cf6ae9-cfd6-4e37-bf3e-e8ee49b54826 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.339681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cf6ae9-cfd6-4e37-bf3e-e8ee49b54826 None admin] PATH_INFO: `/v3/projects/3e92d0c573144f7986682c15b644dc35` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.340800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cf6ae9-cfd6-4e37-bf3e-e8ee49b54826 None admin] RBAC: Authorizing `identity:get_project(project_id=3e92d0c573144f7986682c15b644dc35)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.342234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cf6ae9-cfd6-4e37-bf3e-e8ee49b54826 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.343863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 223/449] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:11 2022] GET /identity/v3/projects/3e92d0c573144f7986682c15b644dc35 => generated 302 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:11.349371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cf6ae9-cfd6-4e37-bf3e-e8ee49b54826 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:11.352310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:11.361786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 227/450] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:27:11.362835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85041d66-480e-425f-89f9-00e6649df94a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:11.430031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:11.450114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cf6ae9-cfd6-4e37-bf3e-e8ee49b54826 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:11.469174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85041d66-480e-425f-89f9-00e6649df94a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:11.490722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d373dcd4-75d0-4ffe-a902-b67c050436e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.491695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373dcd4-75d0-4ffe-a902-b67c050436e8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.492041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373dcd4-75d0-4ffe-a902-b67c050436e8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.492224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373dcd4-75d0-4ffe-a902-b67c050436e8 None admin] PATH_INFO: `/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.493452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d373dcd4-75d0-4ffe-a902-b67c050436e8 None admin] RBAC: Authorizing `identity:get_group(group_id=6d98568fe8684cffb64ab94bdcc0f55e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.494977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d373dcd4-75d0-4ffe-a902-b67c050436e8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.496556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 224/451] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:11 2022] GET /identity/v3/groups/6d98568fe8684cffb64ab94bdcc0f55e => generated 226 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:11.503512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d373dcd4-75d0-4ffe-a902-b67c050436e8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:11.518808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59111545-dba9-451e-a3b6-18bdabde7f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.519459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59111545-dba9-451e-a3b6-18bdabde7f49 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.519741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59111545-dba9-451e-a3b6-18bdabde7f49 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.520022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59111545-dba9-451e-a3b6-18bdabde7f49 None admin] PATH_INFO: `/v3/projects/fabab14479bd4d4c870f8f76efa1b499` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.521204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59111545-dba9-451e-a3b6-18bdabde7f49 None admin] RBAC: Authorizing `identity:get_project(project_id=fabab14479bd4d4c870f8f76efa1b499)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.523678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59111545-dba9-451e-a3b6-18bdabde7f49 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.524961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 228/452] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:27:11 2022] GET /identity/v3/projects/fabab14479bd4d4c870f8f76efa1b499 => generated 344 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:11.531288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59111545-dba9-451e-a3b6-18bdabde7f49 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:11.610508 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d373dcd4-75d0-4ffe-a902-b67c050436e8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:11.628244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:11.652338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.654694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.654694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.654694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.654694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] RBAC: Adding query filter params (group.id=6d98568fe8684cffb64ab94bdcc0f55e, role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=3e92d0c573144f7986682c15b644dc35) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:11.654694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.656419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.663261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 225/453] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:11 2022] GET /identity/v3/role_assignments?group.id=6d98568fe8684cffb64ab94bdcc0f55e&scope.project.id=3e92d0c573144f7986682c15b644dc35&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:11.674959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59111545-dba9-451e-a3b6-18bdabde7f49 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:11.715166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438ce1ed-1c65-404d-b30d-21b55ec16740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.715800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ce1ed-1c65-404d-b30d-21b55ec16740 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.716081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ce1ed-1c65-404d-b30d-21b55ec16740 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.716353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ce1ed-1c65-404d-b30d-21b55ec16740 None admin] PATH_INFO: `/v3/groups/15cf8888d7c44182ba2e623676fc9f7d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.717520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ce1ed-1c65-404d-b30d-21b55ec16740 None admin] RBAC: Authorizing `identity:get_group(group_id=15cf8888d7c44182ba2e623676fc9f7d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.719031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ce1ed-1c65-404d-b30d-21b55ec16740 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.720735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 229/454] 10.222.0.77 () {56 vars in 1180 bytes} [Mon Aug 29 22:27:11 2022] GET /identity/v3/groups/15cf8888d7c44182ba2e623676fc9f7d => generated 229 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:11.727053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:11.743989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:11.848295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7102a9c8-8c2b-4c38-8353-d9b2009185e3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:11.887862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279f02db-5027-4343-8282-534f5cf2321f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:11.888470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279f02db-5027-4343-8282-534f5cf2321f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:11.888745 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279f02db-5027-4343-8282-534f5cf2321f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:11.889012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279f02db-5027-4343-8282-534f5cf2321f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:11.889848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279f02db-5027-4343-8282-534f5cf2321f None admin] RBAC: Adding query filter params (group.id=15cf8888d7c44182ba2e623676fc9f7d, role.id=38171c89968943bd92d9e9758db65615, scope.project.id=fabab14479bd4d4c870f8f76efa1b499) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:11.890157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279f02db-5027-4343-8282-534f5cf2321f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:11.891625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279f02db-5027-4343-8282-534f5cf2321f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:11.900300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 226/455] 10.222.0.77 () {56 vars in 1376 bytes} [Mon Aug 29 22:27:11 2022] GET /identity/v3/role_assignments?group.id=15cf8888d7c44182ba2e623676fc9f7d&scope.project.id=fabab14479bd4d4c870f8f76efa1b499&role.id=38171c89968943bd92d9e9758db65615 => generated 611 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:14.699711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044a0ac-c059-4afa-9a26-908fdcce1239 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:14.700815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044a0ac-c059-4afa-9a26-908fdcce1239 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:14.701291 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044a0ac-c059-4afa-9a26-908fdcce1239 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:14.703247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 230/456] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:14.704906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ca9a0a-fdca-4bc7-b323-a75b05d481c2 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:14.705691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ca9a0a-fdca-4bc7-b323-a75b05d481c2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:14.706178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ca9a0a-fdca-4bc7-b323-a75b05d481c2 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:14.709057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 227/457] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:14.713842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35709c40-7c0e-4cf9-9e5c-a07adc88a255 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:14.714396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35709c40-7c0e-4cf9-9e5c-a07adc88a255 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:14.714906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35709c40-7c0e-4cf9-9e5c-a07adc88a255 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:14.722552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4236fbe-8400-4197-a823-d37d39ed9682 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:14.723670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4236fbe-8400-4197-a823-d37d39ed9682 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:14.724948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4236fbe-8400-4197-a823-d37d39ed9682 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:14.750524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-35709c40-7c0e-4cf9-9e5c-a07adc88a255 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:14.752105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-35709c40-7c0e-4cf9-9e5c-a07adc88a255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:14.762056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b4236fbe-8400-4197-a823-d37d39ed9682 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:14.768263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b4236fbe-8400-4197-a823-d37d39ed9682 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:14.864607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 228/458] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:14.866479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99531a2-e21a-4f5e-a639-526b0ae3c9e1 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:14.866997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99531a2-e21a-4f5e-a639-526b0ae3c9e1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:14.867546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99531a2-e21a-4f5e-a639-526b0ae3c9e1 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:14.869378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 229/459] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:14.873307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 231/460] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:14.880223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caae3902-7d17-4142-9469-5fdc2ee51c4a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:14.880753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caae3902-7d17-4142-9469-5fdc2ee51c4a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:14.881345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caae3902-7d17-4142-9469-5fdc2ee51c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:14.897633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:14.909330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-caae3902-7d17-4142-9469-5fdc2ee51c4a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:14.910922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-caae3902-7d17-4142-9469-5fdc2ee51c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:15.012643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 230/461] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:15.070240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be01c0c8-54bd-4c57-ae50-7038bc1d7bb7 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:15.070932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be01c0c8-54bd-4c57-ae50-7038bc1d7bb7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:15.071616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be01c0c8-54bd-4c57-ae50-7038bc1d7bb7 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:15.074287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 232/462] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:15.089740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b080ec1f-1318-4837-b71f-f98d71b3d34b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:15.090574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b080ec1f-1318-4837-b71f-f98d71b3d34b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:15.091380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b080ec1f-1318-4837-b71f-f98d71b3d34b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:15.131080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b080ec1f-1318-4837-b71f-f98d71b3d34b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:15.133490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b080ec1f-1318-4837-b71f-f98d71b3d34b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:15.221644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be01c0c8-54bd-4c57-ae50-7038bc1d7bb7 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:15.291364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 231/463] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:15.293395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b080ec1f-1318-4837-b71f-f98d71b3d34b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:15.349662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:15.392638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be01c0c8-54bd-4c57-ae50-7038bc1d7bb7 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:15.435036 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:15.435918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:15.436378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:15.436797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:15.439033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:15.452502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:15.454652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:15.474141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:15.475130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:15.476025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:15.476025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:15.476025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:15.477097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 233/464] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:15.479690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:15.509482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b080ec1f-1318-4837-b71f-f98d71b3d34b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:15.552668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:15.553572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:15.554088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:15.554523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:15.566663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:15.568897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:15.586646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:15.587804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:15.588819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:15.588819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:15.588819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:15.590880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 232/465] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:15.591842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:15.592265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418b983-4077-492b-b275-0ea1e5ea541d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:15.619820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:15.645131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:15.645775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:15.646305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:15.646642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:15.658897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:15.660435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:15.675029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:15.675994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:15.676893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:15.676893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:15.676893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:15.676893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 234/466] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:15.677120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:15.703739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1bb29f-dc81-4f41-8b80-b03d11a2894a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:15.723598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:15.743334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:15.743940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:15.744217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:15.744485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:15.745126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:15.746017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:15.747442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:15.760168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 233/467] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:15.761314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:15.804830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e937a68b-1e9a-44aa-82f7-9ca8f3fa4a07 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:15.843656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:15.844271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:15.844554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:15.844829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:15.845592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:15.845820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:15.847270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:15.865037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 235/468] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:15.865863 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:15.871974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72dcd2b-f696-4ff4-8a2e-fd79eac6f3fa None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:15.895859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:15.906896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:15.907750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:15.908238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:15.908713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:15.920618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:15.921701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:15.937374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:15.938449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:15.939618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:15.939618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:15.939618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:15.939618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 234/469] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:15.939618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:15.969238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:15.971822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d407cff6-7e71-40ab-be0d-00855a30ce85 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:16.010974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdc7c10-f0ba-4ea2-94c3-760d43a0b20e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:16.011594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc7c10-f0ba-4ea2-94c3-760d43a0b20e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.011871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc7c10-f0ba-4ea2-94c3-760d43a0b20e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.012174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc7c10-f0ba-4ea2-94c3-760d43a0b20e None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.012807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdc7c10-f0ba-4ea2-94c3-760d43a0b20e None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:16.013113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdc7c10-f0ba-4ea2-94c3-760d43a0b20e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:16.014526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdc7c10-f0ba-4ea2-94c3-760d43a0b20e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:16.026412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 236/470] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:16.027735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad551a17-9b36-4ef5-a9ac-4f792589a76a None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.028035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad551a17-9b36-4ef5-a9ac-4f792589a76a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.028297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad551a17-9b36-4ef5-a9ac-4f792589a76a None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.031891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 237/471] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:16.031891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad551a17-9b36-4ef5-a9ac-4f792589a76a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:16.045367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1e59bc-0974-435b-9cd4-f5c7a1cbcdf7 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:16.080734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e0fc38-12af-4ecb-8d23-4a815f960f50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:16.081255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e0fc38-12af-4ecb-8d23-4a815f960f50 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.081562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e0fc38-12af-4ecb-8d23-4a815f960f50 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.081885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e0fc38-12af-4ecb-8d23-4a815f960f50 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.082545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e0fc38-12af-4ecb-8d23-4a815f960f50 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:16.084048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e0fc38-12af-4ecb-8d23-4a815f960f50 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:16.105860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 235/472] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:27:15 2022] POST /identity/v3/users => generated 248 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:16.106696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e0fc38-12af-4ecb-8d23-4a815f960f50 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:16.156524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:16.174230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad551a17-9b36-4ef5-a9ac-4f792589a76a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:16.216464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c280c97-cf1c-4c86-9ca7-6f35358ca71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:16.217093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c280c97-cf1c-4c86-9ca7-6f35358ca71a None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.217331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:16.217477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c280c97-cf1c-4c86-9ca7-6f35358ca71a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.217644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c280c97-cf1c-4c86-9ca7-6f35358ca71a None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.218289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c280c97-cf1c-4c86-9ca7-6f35358ca71a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:16.219841 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c280c97-cf1c-4c86-9ca7-6f35358ca71a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:16.245489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 238/473] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:27:16 2022] POST /identity/v3/users => generated 246 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:16.246387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c280c97-cf1c-4c86-9ca7-6f35358ca71a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:16.248201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e0fc38-12af-4ecb-8d23-4a815f960f50 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:16.309342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33cb92c-d3bb-44f7-8be8-9a10c037bf88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:16.309993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cb92c-d3bb-44f7-8be8-9a10c037bf88 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.310277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cb92c-d3bb-44f7-8be8-9a10c037bf88 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.310772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cb92c-d3bb-44f7-8be8-9a10c037bf88 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.311275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33cb92c-d3bb-44f7-8be8-9a10c037bf88 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:16.311588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33cb92c-d3bb-44f7-8be8-9a10c037bf88 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:16.313161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33cb92c-d3bb-44f7-8be8-9a10c037bf88 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:16.328474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 236/474] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:16.329834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345df0d7-0db9-4f9c-b422-56e0ab36375e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.329834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345df0d7-0db9-4f9c-b422-56e0ab36375e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.330171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345df0d7-0db9-4f9c-b422-56e0ab36375e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.357511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-345df0d7-0db9-4f9c-b422-56e0ab36375e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:16.358702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-345df0d7-0db9-4f9c-b422-56e0ab36375e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:16.417817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c280c97-cf1c-4c86-9ca7-6f35358ca71a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:16.451663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 237/475] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:16.452848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345df0d7-0db9-4f9c-b422-56e0ab36375e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:16.463884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda5b5f4-1faa-4250-aaa7-3f671d7987b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:16.465350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda5b5f4-1faa-4250-aaa7-3f671d7987b6 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.465459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda5b5f4-1faa-4250-aaa7-3f671d7987b6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.465747 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda5b5f4-1faa-4250-aaa7-3f671d7987b6 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.466414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda5b5f4-1faa-4250-aaa7-3f671d7987b6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:16.468313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda5b5f4-1faa-4250-aaa7-3f671d7987b6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:16.497728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:16.509485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 239/476] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:27:16 2022] POST /identity/v3/users => generated 248 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:16.565698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:16.596374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345df0d7-0db9-4f9c-b422-56e0ab36375e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:16.637652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5530150-1b10-4bdb-a9c4-30ca9ddb3f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:16.638530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5530150-1b10-4bdb-a9c4-30ca9ddb3f38 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.638990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5530150-1b10-4bdb-a9c4-30ca9ddb3f38 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5530150-1b10-4bdb-a9c4-30ca9ddb3f38 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.640275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5530150-1b10-4bdb-a9c4-30ca9ddb3f38 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:16.641937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5530150-1b10-4bdb-a9c4-30ca9ddb3f38 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:16.670554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 238/477] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:27:16 2022] POST /identity/v3/users => generated 249 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:16.754517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda5b5f4-1faa-4250-aaa7-3f671d7987b6 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:16.880906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda5b5f4-1faa-4250-aaa7-3f671d7987b6 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:16.922462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:16.923116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:16.923398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:16.923669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:16.924398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:16.925795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:16.936121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.sql.core [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 29 22:27:16.936121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 29 22:27:16.936121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [parameters: {'id': '80f1a28071bc45b1aee742d4089fe42e', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 29 22:27:16.936121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=103002) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application conn.query(q) Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result.read() Aug 29 22:27:16.953602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return ref.to_dict() Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 22:27:16.954512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application next(self.gen) Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application yield resource Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application next(self.gen) Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.session.rollback() Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.force_reraise() Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise self.value Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application session.commit() Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 29 22:27:16.955475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._prepare_impl() Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.session.flush() Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._flush(objects) Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application compat.raise_( Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application flush_context.execute() Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rec.execute(self) Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 29 22:27:16.956484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 22:27:16.957547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application conn.query(q) Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result.read() Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 22:27:16.958562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [parameters: {'id': '80f1a28071bc45b1aee742d4089fe42e', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:16.959578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 29 22:27:16.960559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:16.961378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 240/478] 10.222.0.77 () {60 vars in 1131 bytes} [Mon Aug 29 22:27:16 2022] POST /identity/v3/roles => generated 144 bytes in 202 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:16.961441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5530150-1b10-4bdb-a9c4-30ca9ddb3f38 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:17.087190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5530150-1b10-4bdb-a9c4-30ca9ddb3f38 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:17.120874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:17.124478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e932f62-26f4-4a49-8364-6c1affaab575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:17.125132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e932f62-26f4-4a49-8364-6c1affaab575 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:17.125412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e932f62-26f4-4a49-8364-6c1affaab575 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:17.125685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e932f62-26f4-4a49-8364-6c1affaab575 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:17.137750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e932f62-26f4-4a49-8364-6c1affaab575 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-0e932f62-26f4-4a49-8364-6c1affaab575 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:17.145450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:17.147583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 29 22:27:17.149510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:17.154779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 239/479] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:27:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:17.155046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:17.221512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:17.264318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258de383-59e8-4d0e-aa83-f23329784721 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:17.302659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5e1300-01d1-4d78-92f7-cdce510a6c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:17.303374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5e1300-01d1-4d78-92f7-cdce510a6c05 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:17.303663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5e1300-01d1-4d78-92f7-cdce510a6c05 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:17.303939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5e1300-01d1-4d78-92f7-cdce510a6c05 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:17.304619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5e1300-01d1-4d78-92f7-cdce510a6c05 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:17.304926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5e1300-01d1-4d78-92f7-cdce510a6c05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:17.306324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5e1300-01d1-4d78-92f7-cdce510a6c05 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:17.316504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 241/480] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:18.688707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5555b-f329-4394-98f7-64d192f368fd None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:18.689259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5555b-f329-4394-98f7-64d192f368fd None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:18.689259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5555b-f329-4394-98f7-64d192f368fd None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:18.690844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 240/481] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:18.696158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f29a59-97e5-4d73-ad2f-f942dc34113b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:18.696555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f29a59-97e5-4d73-ad2f-f942dc34113b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:18.696737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f29a59-97e5-4d73-ad2f-f942dc34113b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:18.723829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-77f29a59-97e5-4d73-ad2f-f942dc34113b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:18.725038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-77f29a59-97e5-4d73-ad2f-f942dc34113b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:18.833603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 242/482] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:18.866314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dad1a8-c918-4a8d-b896-1300d6e487a5 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:18.866847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dad1a8-c918-4a8d-b896-1300d6e487a5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:18.867261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dad1a8-c918-4a8d-b896-1300d6e487a5 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:18.868953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 241/483] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:18.874671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4582ee-283b-44de-a707-3217c3695c27 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:18.875184 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4582ee-283b-44de-a707-3217c3695c27 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:18.875610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4582ee-283b-44de-a707-3217c3695c27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:18.907769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7a4582ee-283b-44de-a707-3217c3695c27 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:18.909195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7a4582ee-283b-44de-a707-3217c3695c27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:18.987300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:19.009057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 243/484] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:19.224232 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dad1a8-c918-4a8d-b896-1300d6e487a5 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:19.231066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4a1e58-018c-409e-91ae-1a1fc01c197f None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:19.231614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4a1e58-018c-409e-91ae-1a1fc01c197f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:19.232084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4a1e58-018c-409e-91ae-1a1fc01c197f None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:19.233866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 244/485] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:19.258859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeff73f-3648-4469-8390-9740ea58aa5e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:19.259410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeff73f-3648-4469-8390-9740ea58aa5e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:19.259900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeff73f-3648-4469-8390-9740ea58aa5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:19.299749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-0aeff73f-3648-4469-8390-9740ea58aa5e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:19.301301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0aeff73f-3648-4469-8390-9740ea58aa5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:19.429006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dad1a8-c918-4a8d-b896-1300d6e487a5 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:19.434673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 245/486] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:19.435813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aeff73f-3648-4469-8390-9740ea58aa5e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:19.535608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc2b3f9-0560-413c-8a8d-16000c7f797a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:19.536186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc2b3f9-0560-413c-8a8d-16000c7f797a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:19.536473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc2b3f9-0560-413c-8a8d-16000c7f797a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:19.536751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc2b3f9-0560-413c-8a8d-16000c7f797a None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:19.559349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aeff73f-3648-4469-8390-9740ea58aa5e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:19.565635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc2b3f9-0560-413c-8a8d-16000c7f797a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-acc2b3f9-0560-413c-8a8d-16000c7f797a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:19.571095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:19.572003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:19.572829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 242/487] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:19.574189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25facb90-170e-4654-b1b0-774a44a8c91e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:19.574861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25facb90-170e-4654-b1b0-774a44a8c91e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:19.575525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25facb90-170e-4654-b1b0-774a44a8c91e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:19.577011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 243/488] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:19.582413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ceaa01-65fc-4000-a2a6-7eebf410497b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:19.582888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ceaa01-65fc-4000-a2a6-7eebf410497b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:19.583366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ceaa01-65fc-4000-a2a6-7eebf410497b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:19.584770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:19.606093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-17ceaa01-65fc-4000-a2a6-7eebf410497b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:19.607761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-17ceaa01-65fc-4000-a2a6-7eebf410497b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:19.609912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49c61af-22e7-4a71-a1e0-a255dfcff56c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:19.611017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49c61af-22e7-4a71-a1e0-a255dfcff56c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:19.612424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49c61af-22e7-4a71-a1e0-a255dfcff56c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:19.612970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49c61af-22e7-4a71-a1e0-a255dfcff56c None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:19.627913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49c61af-22e7-4a71-a1e0-a255dfcff56c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-c49c61af-22e7-4a71-a1e0-a255dfcff56c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:19.634463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:19.635422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:19.636783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:19.639413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 246/489] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:19.640061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49c61af-22e7-4a71-a1e0-a255dfcff56c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:19.680823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:19.717227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 244/490] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:19.718943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ceaa01-65fc-4000-a2a6-7eebf410497b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:19.798289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49c61af-22e7-4a71-a1e0-a255dfcff56c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:19.841486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:19.843175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:19.843668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:19.844145 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:19.844785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ceaa01-65fc-4000-a2a6-7eebf410497b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:19.845126 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] RBAC: Adding query filter params (name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:19.845536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:19.847320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:19.859287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 247/491] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:19.860976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:19.899344 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:19.900256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:19.900543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:19.900815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:19.901503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:19.902049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:19.905322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:19.915334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:19.922903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 245/492] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:19.925416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:19.977890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:20.006104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737e958f-084e-42de-85d1-10b9c5b2b2e6 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.039608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1640fc50-3dd9-43ce-b3a3-6b2af71b7a07 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.044799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ff17cc-4683-4dfc-8696-24874a8edf6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.045435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff17cc-4683-4dfc-8696-24874a8edf6f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.045719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff17cc-4683-4dfc-8696-24874a8edf6f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.045988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff17cc-4683-4dfc-8696-24874a8edf6f None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.056949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ff17cc-4683-4dfc-8696-24874a8edf6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-19ff17cc-4683-4dfc-8696-24874a8edf6f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:20.062830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:20.063735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:20.064544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 248/493] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:20.065209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ff17cc-4683-4dfc-8696-24874a8edf6f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.078450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.079111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.079393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.079663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.090462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.092076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:20.106463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:20.107458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:20.108314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:20.108314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:20.108314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:20.108314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 246/494] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:20.108542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.167228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:20.171872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ff17cc-4683-4dfc-8696-24874a8edf6f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.210372 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4375e305-e523-428e-94fc-70b3b4501801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.211041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375e305-e523-428e-94fc-70b3b4501801 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.211328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375e305-e523-428e-94fc-70b3b4501801 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.211604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375e305-e523-428e-94fc-70b3b4501801 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.211868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:20.219813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5a7f3-7f8d-4fce-b1c2-21b612167586 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.223787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375e305-e523-428e-94fc-70b3b4501801 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.225269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375e305-e523-428e-94fc-70b3b4501801 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-4375e305-e523-428e-94fc-70b3b4501801 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:20.240153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:20.241094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:20.241950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:20.241950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:20.241950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:20.241950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 249/495] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:20.242512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1351674b-85e8-493f-8d09-3c8a3a178695 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.242755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1351674b-85e8-493f-8d09-3c8a3a178695 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.243022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1351674b-85e8-493f-8d09-3c8a3a178695 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.244233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 250/496] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:20 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:20.244919 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1351674b-85e8-493f-8d09-3c8a3a178695 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.260568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036e4b17-d8d6-4f3d-a1ec-d340362a7084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.261224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036e4b17-d8d6-4f3d-a1ec-d340362a7084 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.261504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036e4b17-d8d6-4f3d-a1ec-d340362a7084 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.261780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036e4b17-d8d6-4f3d-a1ec-d340362a7084 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.271881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036e4b17-d8d6-4f3d-a1ec-d340362a7084 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-036e4b17-d8d6-4f3d-a1ec-d340362a7084 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:20.277191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:20.278074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:20.279116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 247/497] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:20.284402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036e4b17-d8d6-4f3d-a1ec-d340362a7084 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.350213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1351674b-85e8-493f-8d09-3c8a3a178695 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.385096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.385701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.385800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036e4b17-d8d6-4f3d-a1ec-d340362a7084 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.385987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.386291 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.387266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] RBAC: Adding query filter params (name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:20.387382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.388741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.397078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 251/498] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:20.397716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.413211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:20.425650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a19c43-76bc-4046-9957-cddbcde55e93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.426255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a19c43-76bc-4046-9957-cddbcde55e93 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.426538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a19c43-76bc-4046-9957-cddbcde55e93 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.426848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a19c43-76bc-4046-9957-cddbcde55e93 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.427493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a19c43-76bc-4046-9957-cddbcde55e93 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:20.427798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a19c43-76bc-4046-9957-cddbcde55e93 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.429201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a19c43-76bc-4046-9957-cddbcde55e93 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.441934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 248/499] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:20.443203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5096997-b397-45a8-a377-e8891d369b77 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.443486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5096997-b397-45a8-a377-e8891d369b77 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.443758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5096997-b397-45a8-a377-e8891d369b77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.462613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f5096997-b397-45a8-a377-e8891d369b77 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:20.463866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f5096997-b397-45a8-a377-e8891d369b77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:20.475650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:20.510127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d1246-e168-4b20-9c0f-4fe3109e3a6b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.545614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 249/500] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:20.546567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5096997-b397-45a8-a377-e8891d369b77 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.555466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.556868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.557156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.562996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.564691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:20.565014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.566432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.592670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 252/501] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:20.597818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.672405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5096997-b397-45a8-a377-e8891d369b77 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.683367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:20.710281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.710931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.711208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.711477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.712145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:20.712448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.713833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.718382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce968c-f106-49cb-8637-e0b1f22520e0 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.728089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 250/502] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:20.735661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.748592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:20.766143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.766633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.766929 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.767196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.778954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.780445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:20.799377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:20.800788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:20.801622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:20.801622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:20.801622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:20.801622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 253/503] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:20.801843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.859124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e0138b-04d8-4a2c-9c9d-5b8bb7ab8ba6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.893674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.894304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.894584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.894896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] PATH_INFO: `/v3/users/glance` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.902913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.904372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.910032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b412cf6e-4247-45e6-8102-3a040e843398 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:20.917149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:20.918694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 251/504] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/users/glance => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:20.919638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.936119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:20.947155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:20.947779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:20.948053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:20.948322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] PATH_INFO: `/v3/users/nova` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:20.956750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:20.958343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:20.971971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:20.973113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:20.974043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:20.974808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 254/505] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:20.975746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:20.987541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.023202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e34a5-d427-4e4e-8345-ce56e196930f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.057157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.057782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.058096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.058372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.069039 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.070555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.077681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6687b903-11c0-4f9b-b14a-ffd3a9549703 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:21.085703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:21.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:21.087876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:21.087876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:21.087876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:21.087876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 252/506] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:21.089078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.112219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.112817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.113091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.113357 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.113994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:21.114291 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.115706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.127272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 255/507] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:21.128070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.176991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.191375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51700d6-cd59-42a1-adbb-091fadb64e2e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.211482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.226920 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.227550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.227828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.228097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.233172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a387c-8c12-4271-8108-651c21eb1a8f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.238792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.240266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:21.254878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:21.255798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:21.256650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:21.256650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:21.256650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:21.256650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 253/508] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:21.256870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.270132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.270777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.271065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.271338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.272079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:21.272387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.273829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.313095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 256/509] 10.222.0.77 () {56 vars in 1137 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:21.313945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.364520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6483df95-b391-459f-887a-2fe3ebba6f49 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.398778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.399390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.399667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.399943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.400681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:21.400986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.402478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.417795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2cb3e4-e040-4546-b1e2-2fa15cfe74c7 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.419390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.455153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 254/510] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:21.455960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.457445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.458158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.458539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.458967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.459759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:21.460165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.461657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.474428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 257/511] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:21.475428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.491712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.569022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e83c27e-7bc5-47c8-adb4-c1822bbb452a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.591793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98aa334-c9d7-44de-9773-fe57aead39c6 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.609816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.610448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.610744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.611024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.618287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.619784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.627977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.628582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.628858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.629296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.629778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:21.630141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.631507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:21.632120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:21.633421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 22:27:21.635074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:21.638653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 255/512] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/users/cinder => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:21.639574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.650520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 258/513] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:21.650520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.677389 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.733514 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.743673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3256b5-37f1-4c80-8bd7-2c436891c42a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.757300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e169f6-0d75-4843-926e-1e3e992289cb None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:21.780862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88d9e3a-5cc0-4108-985a-f97a6c967060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.781738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d9e3a-5cc0-4108-985a-f97a6c967060 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.782025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d9e3a-5cc0-4108-985a-f97a6c967060 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.782424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d9e3a-5cc0-4108-985a-f97a6c967060 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.802636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95466628-7f87-4c0d-b93d-55e559c65891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:21.803403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95466628-7f87-4c0d-b93d-55e559c65891 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:21.803800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95466628-7f87-4c0d-b93d-55e559c65891 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:21.804127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95466628-7f87-4c0d-b93d-55e559c65891 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:21.860133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88d9e3a-5cc0-4108-985a-f97a6c967060 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.861634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88d9e3a-5cc0-4108-985a-f97a6c967060 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.884586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95466628-7f87-4c0d-b93d-55e559c65891 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:21.886065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95466628-7f87-4c0d-b93d-55e559c65891 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:21.911117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.957548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:21.969609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 256/514] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:21.969972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88d9e3a-5cc0-4108-985a-f97a6c967060 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:21.997099 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 259/515] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:21.997954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95466628-7f87-4c0d-b93d-55e559c65891 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.073441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88d9e3a-5cc0-4108-985a-f97a6c967060 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.098963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95466628-7f87-4c0d-b93d-55e559c65891 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.108308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.108940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.109221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.109501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.116565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.118113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.121771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:22.130271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:22.131278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 257/516] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/users/neutron => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:22.132196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.133519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.134035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.134309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.134578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.135361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:22.135664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.137659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.154969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 260/517] 10.222.0.77 () {56 vars in 1137 bytes} [Mon Aug 29 22:27:21 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:22.155805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.189724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:22.248584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031810c2-b72a-4057-a6ad-ef4c36bcb226 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.262397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb1287-30ba-40d2-82a9-9d8df8d74765 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.283080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f86f66-be11-4e5c-a32a-cd4a687d9fbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.283784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f86f66-be11-4e5c-a32a-cd4a687d9fbc None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.283968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f86f66-be11-4e5c-a32a-cd4a687d9fbc None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.284238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f86f66-be11-4e5c-a32a-cd4a687d9fbc None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.284894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f86f66-be11-4e5c-a32a-cd4a687d9fbc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.286348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f86f66-be11-4e5c-a32a-cd4a687d9fbc None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.297253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.298186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.298408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.298827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.307449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 258/518] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:27:22 2022] POST /identity/v3/users => generated 247 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:22.309400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f86f66-be11-4e5c-a32a-cd4a687d9fbc None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.310454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.312188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:22.329337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:22.330554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:22.331662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:22.331662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:22.331662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:22.335376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 261/519] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:22.336232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.372306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:22.441267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f86f66-be11-4e5c-a32a-cd4a687d9fbc None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.449400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:22.458355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8deb9b-23c8-43d5-93be-8004303717c8 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.492227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.493002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.494127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.494127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.506214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.507023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.507461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.507636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.507918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.508949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:22.509134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.509438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.511933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:22.524321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:22.526180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:22.528358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:22.528358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:22.528358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:22.528358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 259/520] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:22.531678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.538674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 262/521] 10.222.0.77 () {56 vars in 1139 bytes} [Mon Aug 29 22:27:22 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:22.539720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.642981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8e2a9-069a-479a-9362-008eef3f2495 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.643240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:22.655354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c504dab-cc3b-4871-97d0-f381f5b6d8e3 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.685407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2ac7b8-b521-4bd7-87ee-78de5e24b9d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.686027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2ac7b8-b521-4bd7-87ee-78de5e24b9d6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.686374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2ac7b8-b521-4bd7-87ee-78de5e24b9d6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.686575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2ac7b8-b521-4bd7-87ee-78de5e24b9d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.693199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.693816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.694096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.694369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.695046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:22.695353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.696732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.712999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 263/522] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:22.712999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.720507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:22.765600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2ac7b8-b521-4bd7-87ee-78de5e24b9d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.767149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2ac7b8-b521-4bd7-87ee-78de5e24b9d6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.823571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae610947-bcfd-4092-92e1-bdd721438af4 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:22.865466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:22.866109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:22.866391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:22.866665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:22.867350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:22.867661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:22.869037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:22.881851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 264/523] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:22.882856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.884531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 260/524] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:22.885515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2ac7b8-b521-4bd7-87ee-78de5e24b9d6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:22.900394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:22.953208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:22.995298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c78fd1a-c656-40d5-892c-3de4e5bca6ce None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.005581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2ac7b8-b521-4bd7-87ee-78de5e24b9d6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.034540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d2062-cd4f-4ce7-8ae1-8b430aa48218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.035109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d2062-cd4f-4ce7-8ae1-8b430aa48218 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.035391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d2062-cd4f-4ce7-8ae1-8b430aa48218 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.035684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d2062-cd4f-4ce7-8ae1-8b430aa48218 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.046552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.047224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.047501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.047765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.063032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.064530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:23.078252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:23.079427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:23.080279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 261/525] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:23.080770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.130645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d2062-cd4f-4ce7-8ae1-8b430aa48218 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.132140 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d2062-cd4f-4ce7-8ae1-8b430aa48218 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.143658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:23.194847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ebcbd-4cc2-4673-9f19-24ad44a63f1a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.216434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:23.235875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.236696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.236979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.237260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.242970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 265/526] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:23.243721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d2062-cd4f-4ce7-8ae1-8b430aa48218 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.249093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.250577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:23.263728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:23.264749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 262/527] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:23.265607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.368409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d2062-cd4f-4ce7-8ae1-8b430aa48218 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.380412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:23.389397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b006cd2d-d28e-4664-842b-6631d961852a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.407440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.408081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.408373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.408637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.419419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.420902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.427621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.428237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.428519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.428797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.429500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:23.429810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.431284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:23.437031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:23.438018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:23.439700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:23.439700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:23.439700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:23.439700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 266/528] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:23.439700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.459981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 263/529] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:23.461272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.504525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:23.556400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce598a8-a051-4361-ade1-41b31c1a4049 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.569599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f908fdae-d447-4c3c-8282-c9d846db0766 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.602029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.602663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.602964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.603232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.612992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.613878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.614331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.614516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.614985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.615812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.616074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:23.616549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.618265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:23.630953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:23.631868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:23.632759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:23.632759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:23.632759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:23.635060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 267/530] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:23.635917 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 264/531] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:23.637497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.637648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.656673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:23.745086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:23.749464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c78dc2-6e0f-48cd-b4b3-d0aefc80897f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.750469 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82fd937-3872-49a6-ab9f-c336f1bc7d1c None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.789200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.789787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.790121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.790337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.790771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:23.791252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:23.791328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.791394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:23.791645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:23.791918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:23.792683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=c1e61f3604fd4a23ae5f1e7e62ae21ae) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:23.792782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.792996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:23.794412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:23.800397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 265/532] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/role_assignments?user.id=c1e61f3604fd4a23ae5f1e7e62ae21ae&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:23.801097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.825131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 268/533] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:23.826694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:23.941266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:23.948552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbaad49-83b4-4b8b-a703-3f18d373a98e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:23.974443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b97c55-df51-40e6-a365-064227685d6f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:24.003365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:24.004249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.004669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.005073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.006059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=57491e4de6c3435e9d818266a910b831) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:24.006608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:24.008205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:24.015030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 266/534] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/role_assignments?user.id=57491e4de6c3435e9d818266a910b831&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 259 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:24.016417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:24.018350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:24.019030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.019314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.019589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.020265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:24.020573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:24.023292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:24.039356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 269/535] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:24.048566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:24.062520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:24.208413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a5086f-e07b-4337-98ed-5a0664ad4b9f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:24.214170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0eaeed-2309-412b-8443-368132032ad5 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:24.264291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:24.266926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.267908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.270398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.283591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdc4275-d5d9-49f0-bac6-bda298cdb0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:24.284442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:24.285706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:24.286223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdc4275-d5d9-49f0-bac6-bda298cdb0fb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.286860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdc4275-d5d9-49f0-bac6-bda298cdb0fb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.287319 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdc4275-d5d9-49f0-bac6-bda298cdb0fb None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.290902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:24.301910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdc4275-d5d9-49f0-bac6-bda298cdb0fb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:24.303438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdc4275-d5d9-49f0-bac6-bda298cdb0fb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:24.306370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:24.309421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:24.310571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 267/536] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:24.315033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-bcdc4275-d5d9-49f0-bac6-bda298cdb0fb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:24.329432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:24.330341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 270/537] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:24.332361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108b6ed-01df-4b80-aeb5-55aac9bb35ef None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.333588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108b6ed-01df-4b80-aeb5-55aac9bb35ef None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.333588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108b6ed-01df-4b80-aeb5-55aac9bb35ef None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.335191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 271/538] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:24.338893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e108b6ed-01df-4b80-aeb5-55aac9bb35ef None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:24.426223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:24.444623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0bcecc-c164-4189-b58c-2c4a0e6b8c7e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:24.487381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68741aaa-96e3-44aa-800c-7e88cc08e84c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:24.487988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68741aaa-96e3-44aa-800c-7e88cc08e84c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.488264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68741aaa-96e3-44aa-800c-7e88cc08e84c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.488535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68741aaa-96e3-44aa-800c-7e88cc08e84c None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.489235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68741aaa-96e3-44aa-800c-7e88cc08e84c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:24.489536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68741aaa-96e3-44aa-800c-7e88cc08e84c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:24.489723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e108b6ed-01df-4b80-aeb5-55aac9bb35ef None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:24.491007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68741aaa-96e3-44aa-800c-7e88cc08e84c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:24.504565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 268/539] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:24.505807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aba2efc-4f5a-4a5c-949f-d569e87cab16 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.506087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aba2efc-4f5a-4a5c-949f-d569e87cab16 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.506358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aba2efc-4f5a-4a5c-949f-d569e87cab16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.525229 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5aba2efc-4f5a-4a5c-949f-d569e87cab16 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:24.526423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5aba2efc-4f5a-4a5c-949f-d569e87cab16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:24.539260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:24.540017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.540561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.540841 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.541789 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:24.542103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:24.544428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:24.576725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 272/540] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:24.576725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:24.616399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 269/541] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:24.617645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aba2efc-4f5a-4a5c-949f-d569e87cab16 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:24.645599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:24.705744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6af86f-9135-48b7-8ec5-6e724fb6872f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:24.709486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:24.736306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aba2efc-4f5a-4a5c-949f-d569e87cab16 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:24.744635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:24.745282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.745629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.746580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.747423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=7964b4109f9a4ac6a68d90366489557c) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:24.747797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:24.749392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:24.755517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 273/542] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:24 2022] GET /identity/v3/role_assignments?user.id=7964b4109f9a4ac6a68d90366489557c&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:24.777169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:24.778105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:24.778558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:24.779044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:24.779988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=dea13198ef8e45de85b92e79e3ee71b0) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:24.780509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:24.782468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:24.789489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 270/543] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:24 2022] GET /identity/v3/role_assignments?user.id=dea13198ef8e45de85b92e79e3ee71b0&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:24.975901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:25.104538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9fc258-b395-485c-af16-4ec425a18981 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:25.124186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:25.143168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93733609-bb55-48f1-9792-fa6fbbf64c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:25.143806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93733609-bb55-48f1-9792-fa6fbbf64c51 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:25.144087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93733609-bb55-48f1-9792-fa6fbbf64c51 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:25.144362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93733609-bb55-48f1-9792-fa6fbbf64c51 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:25.157115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93733609-bb55-48f1-9792-fa6fbbf64c51 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-93733609-bb55-48f1-9792-fa6fbbf64c51 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:25.162483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:25.163409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:25.164210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:25.164838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 274/544] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:25.171894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:25.340235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e35055-5ea3-4841-a065-5ba5191e4b79 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:25.392481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:25.393016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:25.393296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:25.393564 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:25.394237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:25.394539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:25.395945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:25.407740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 271/545] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:25.412797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93733609-bb55-48f1-9792-fa6fbbf64c51 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:25.565008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93733609-bb55-48f1-9792-fa6fbbf64c51 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:25.607915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:25.609194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:25.609986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:25.610819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:25.630673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:25.632190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:25.650063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:25.651030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:25.653131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:25.653131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:25.653131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:25.653131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 275/546] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:25.659824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:25.665205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:25.812009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724c531d-c095-48fd-8a1b-ec3200c70619 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:25.864436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:25.865043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:25.865317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:25.865586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:25.866231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:25.866533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:25.867934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:25.887055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 272/547] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:25.892732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:25.948558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:26.068876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43740270-8a8a-447d-9890-7db6fcf2d6c3 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:26.112432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7983bb-60ed-4515-8e44-22110d103a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:26.113071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7983bb-60ed-4515-8e44-22110d103a3b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:26.113349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7983bb-60ed-4515-8e44-22110d103a3b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:26.113622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7983bb-60ed-4515-8e44-22110d103a3b None admin] PATH_INFO: `/v3/users/swift` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:26.122341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7983bb-60ed-4515-8e44-22110d103a3b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:26.123827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7983bb-60ed-4515-8e44-22110d103a3b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-2f7983bb-60ed-4515-8e44-22110d103a3b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:26.135395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:26.137909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:26.139410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 276/548] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:25 2022] GET /identity/v3/users/swift => generated 83 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:26.142999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:26.226110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:26.279580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea6a651-b634-4d1e-bb3c-1c4fca5b698f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:26.285122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be159ae-baf8-45d8-a74b-6273fc6d6e38 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:26.285410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be159ae-baf8-45d8-a74b-6273fc6d6e38 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:26.285693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be159ae-baf8-45d8-a74b-6273fc6d6e38 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:26.287226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 277/549] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:27:26.298632 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685b614-dd83-4a59-a3f4-41adf1b89851 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:26.299285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685b614-dd83-4a59-a3f4-41adf1b89851 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:26.299786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685b614-dd83-4a59-a3f4-41adf1b89851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:26.318954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8685b614-dd83-4a59-a3f4-41adf1b89851 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:26.320521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8685b614-dd83-4a59-a3f4-41adf1b89851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:26.344457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d870e12f-2be9-4744-9092-3759cd81a6ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:26.345309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d870e12f-2be9-4744-9092-3759cd81a6ba None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:26.345815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d870e12f-2be9-4744-9092-3759cd81a6ba None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:26.346307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d870e12f-2be9-4744-9092-3759cd81a6ba None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:26.347254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d870e12f-2be9-4744-9092-3759cd81a6ba None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:26.347570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d870e12f-2be9-4744-9092-3759cd81a6ba None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:26.349159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d870e12f-2be9-4744-9092-3759cd81a6ba None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:26.385838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 273/550] 10.222.0.77 () {56 vars in 1135 bytes} [Mon Aug 29 22:27:26 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:26.387247 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86eca4-e2cc-471a-98b1-e68f1eed71d8 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:26.387540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86eca4-e2cc-471a-98b1-e68f1eed71d8 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:26.387813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86eca4-e2cc-471a-98b1-e68f1eed71d8 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:26.389063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 274/551] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:26.393797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed86eca4-e2cc-471a-98b1-e68f1eed71d8 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:26.428994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 278/552] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:26.430293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac55dbf-0992-4c0b-8715-bc5ce60cf14c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:26.430579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac55dbf-0992-4c0b-8715-bc5ce60cf14c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:26.430927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac55dbf-0992-4c0b-8715-bc5ce60cf14c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:26.450466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2ac55dbf-0992-4c0b-8715-bc5ce60cf14c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:26.451743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2ac55dbf-0992-4c0b-8715-bc5ce60cf14c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:26.467211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:26.530520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 279/553] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:26.531021 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed86eca4-e2cc-471a-98b1-e68f1eed71d8 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:26.553399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:26.570279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0661ae33-761d-4b56-8cbf-592ad0860503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:26.571307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0661ae33-761d-4b56-8cbf-592ad0860503 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:26.571799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0661ae33-761d-4b56-8cbf-592ad0860503 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:26.572264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0661ae33-761d-4b56-8cbf-592ad0860503 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:26.642465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0661ae33-761d-4b56-8cbf-592ad0860503 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:26.644314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0661ae33-761d-4b56-8cbf-592ad0860503 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:26.746766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 275/554] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:26.754517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac55dbf-0992-4c0b-8715-bc5ce60cf14c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:26.829992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0661ae33-761d-4b56-8cbf-592ad0860503 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:26.853524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:26.906085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac55dbf-0992-4c0b-8715-bc5ce60cf14c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:26.944991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:26.945637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:26.946031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:26.946309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:26.953611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0661ae33-761d-4b56-8cbf-592ad0860503 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:26.958412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:26.959910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:26.977093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:26.978077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:26.978986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:26.978986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:26.978986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:26.978986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 280/555] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:26.979619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:26.983133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:27.006063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e18e3c-7985-4d4e-be28-8a5df713b238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.007023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18e3c-7985-4d4e-be28-8a5df713b238 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.007314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18e3c-7985-4d4e-be28-8a5df713b238 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.007589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18e3c-7985-4d4e-be28-8a5df713b238 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.021684 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e18e3c-7985-4d4e-be28-8a5df713b238 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-38e18e3c-7985-4d4e-be28-8a5df713b238 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:27.028897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:27.029806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:27.030633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:27.035204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 276/556] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:27.036524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e18e3c-7985-4d4e-be28-8a5df713b238 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.111613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311ea798-ed41-4469-a22e-097f9443847d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.134402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:27.149575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b67c1d0-5c88-4727-af0f-2abba89c635d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.150573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67c1d0-5c88-4727-af0f-2abba89c635d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.151081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67c1d0-5c88-4727-af0f-2abba89c635d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.151555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b67c1d0-5c88-4727-af0f-2abba89c635d None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.157774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e18e3c-7985-4d4e-be28-8a5df713b238 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.164298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b67c1d0-5c88-4727-af0f-2abba89c635d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-2b67c1d0-5c88-4727-af0f-2abba89c635d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:27.170394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:27.172105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:27.172962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:27.180176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 281/557] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:27.181310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b67c1d0-5c88-4727-af0f-2abba89c635d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.195416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.195964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.196245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.196518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.197176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:27.197484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:27.199440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:27.212192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 277/558] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:27.213122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.252252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:27.296515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b67c1d0-5c88-4727-af0f-2abba89c635d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.325529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25a2a67-aa2f-4d03-b47e-0c8c34c5941d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.338927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.339913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.340533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.341095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.342052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] RBAC: Adding query filter params (name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:27.342539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:27.344302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:27.353736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 282/559] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:27.354933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.367012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b3b9df-dc03-4fa4-8d94-99049403ccd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.367636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b3b9df-dc03-4fa4-8d94-99049403ccd6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.367921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b3b9df-dc03-4fa4-8d94-99049403ccd6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.368194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b3b9df-dc03-4fa4-8d94-99049403ccd6 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.368865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b3b9df-dc03-4fa4-8d94-99049403ccd6 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:27.369187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b3b9df-dc03-4fa4-8d94-99049403ccd6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:27.370541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b3b9df-dc03-4fa4-8d94-99049403ccd6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:27.384370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 278/560] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:27.384370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d21b75-5375-44be-a5b9-e9d728d2b95b None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.384370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d21b75-5375-44be-a5b9-e9d728d2b95b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.384370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d21b75-5375-44be-a5b9-e9d728d2b95b None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.384370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 279/561] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:27.384370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d21b75-5375-44be-a5b9-e9d728d2b95b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.388637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:27.478983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbb40ab-4782-4aec-86b8-d1c746bff8d1 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.510620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:27.511185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d21b75-5375-44be-a5b9-e9d728d2b95b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.522961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.523597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.523880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.524150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.530986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:27.532699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:27.544014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:27.545027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 283/562] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:27.545909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.551865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880c7cf4-0bc5-4f3d-aba5-3c7f45f6a1d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.552500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880c7cf4-0bc5-4f3d-aba5-3c7f45f6a1d0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.552778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880c7cf4-0bc5-4f3d-aba5-3c7f45f6a1d0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.553055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880c7cf4-0bc5-4f3d-aba5-3c7f45f6a1d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.565138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880c7cf4-0bc5-4f3d-aba5-3c7f45f6a1d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:27.566594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880c7cf4-0bc5-4f3d-aba5-3c7f45f6a1d0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-880c7cf4-0bc5-4f3d-aba5-3c7f45f6a1d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:27.581403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:27.582371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:27.583306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:27.583306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:27.583306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:27.583306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 280/563] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:27.584010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32d018d-c6f9-4bca-8af9-74330bdf44b4 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.584010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32d018d-c6f9-4bca-8af9-74330bdf44b4 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.584271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32d018d-c6f9-4bca-8af9-74330bdf44b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.601989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d32d018d-c6f9-4bca-8af9-74330bdf44b4 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:27.603234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d32d018d-c6f9-4bca-8af9-74330bdf44b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:27.653783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde393c-740d-4161-802a-2ec1b6640c11 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.655714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:27.674201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 281/564] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:27.675571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def0c655-f7df-4823-a671-0e07988dc107 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.675868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def0c655-f7df-4823-a671-0e07988dc107 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.676136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def0c655-f7df-4823-a671-0e07988dc107 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.678768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 282/565] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:27.678768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def0c655-f7df-4823-a671-0e07988dc107 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.689681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.690322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.690582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.691832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.702649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:27.704154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:27.718396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:27.719590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:27.720664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:27.720664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:27.720664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:27.720664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 284/566] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:27.720664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.761493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:27.812297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def0c655-f7df-4823-a671-0e07988dc107 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.837531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0624448-baeb-4a18-af93-4b115c945dc9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:27.855725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fbd0a6-86fd-46f9-b676-bc2683d38bb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.856703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fbd0a6-86fd-46f9-b676-bc2683d38bb1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.857385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fbd0a6-86fd-46f9-b676-bc2683d38bb1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.857834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fbd0a6-86fd-46f9-b676-bc2683d38bb1 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.858683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fbd0a6-86fd-46f9-b676-bc2683d38bb1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:27.859151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fbd0a6-86fd-46f9-b676-bc2683d38bb1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:27.861476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fbd0a6-86fd-46f9-b676-bc2683d38bb1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:27.874498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:27.875133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.875413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.875685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.876332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:27.876734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:27.877116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 283/567] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:27.878172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:27.878656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d02f1-ae11-4cf2-bc68-ec5c3d18f2a2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:27.879088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d02f1-ae11-4cf2-bc68-ec5c3d18f2a2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:27.879771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d02f1-ae11-4cf2-bc68-ec5c3d18f2a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:27.891508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 285/568] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:27.892414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:27.905750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2a8d02f1-ae11-4cf2-bc68-ec5c3d18f2a2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:27.907818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2a8d02f1-ae11-4cf2-bc68-ec5c3d18f2a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:27.956637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:27.984172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 284/569] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:27.985008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d02f1-ae11-4cf2-bc68-ec5c3d18f2a2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.005827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:28.013900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991d0cd-1afc-4df4-825d-050566929c5d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.054881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.055493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.055772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.056047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.056689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:28.056992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:28.058921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:28.073852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 286/570] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:28.076313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.094650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d02f1-ae11-4cf2-bc68-ec5c3d18f2a2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.132404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.133002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.133302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.133562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.134283 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=03361f4b51d34cf28c4b5451c3b1f881) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:28.134602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:28.136053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:28.142250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 285/571] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:27 2022] GET /identity/v3/role_assignments?user.id=03361f4b51d34cf28c4b5451c3b1f881&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:28.143073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.190599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:28.212894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10289d30-17d3-474e-ac01-76e7df7bad5f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.254993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.255939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.256053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.256310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] PATH_INFO: `/v3/users/glance` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.266191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:28.267719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:28.279735 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8007cd5d-7f61-43df-8879-e76cfb21af42 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:28.282027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:28.285454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 287/572] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/users/glance => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:28.286471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.305321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:28.325561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48c78c-7686-42a6-b0ba-4f0ebbe25758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.326284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48c78c-7686-42a6-b0ba-4f0ebbe25758 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.326635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48c78c-7686-42a6-b0ba-4f0ebbe25758 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.327007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48c78c-7686-42a6-b0ba-4f0ebbe25758 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.339047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a48c78c-7686-42a6-b0ba-4f0ebbe25758 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-5a48c78c-7686-42a6-b0ba-4f0ebbe25758 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:28.345650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:28.346690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 286/573] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:28.348159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48c78c-7686-42a6-b0ba-4f0ebbe25758 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.396786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b5083d-bbe4-49ac-b4e0-1ac8c0d43338 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.436867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.437495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.437774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.438340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] PATH_INFO: `/v3/users/nova` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.445782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:28.447218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:28.459177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:28.460271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 288/574] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:28.461221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.466603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:28.471061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48c78c-7686-42a6-b0ba-4f0ebbe25758 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.513022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c44603-dd7b-443e-a555-d9e623e9ad87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.513661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c44603-dd7b-443e-a555-d9e623e9ad87 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.513943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c44603-dd7b-443e-a555-d9e623e9ad87 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.514221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c44603-dd7b-443e-a555-d9e623e9ad87 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.524766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c44603-dd7b-443e-a555-d9e623e9ad87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-c8c44603-dd7b-443e-a555-d9e623e9ad87 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:28.530766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:28.532157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 287/575] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:28.532989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c44603-dd7b-443e-a555-d9e623e9ad87 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.554902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:28.582167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef27c834-4405-48ae-ba0a-21bb7b6727aa None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.621858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.622474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.622785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.623071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.623814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:28.624127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:28.625570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:28.645888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 289/576] 10.222.0.77 () {56 vars in 1137 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:28.646830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.647390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c44603-dd7b-443e-a555-d9e623e9ad87 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.687094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.687706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.687984 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.688251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.688923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:28.689228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:28.690599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:28.699712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 288/577] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:28.700448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.723853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:28.765875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d984e374-9af6-4a34-a17b-28d013179c57 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.805552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.806166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.806448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.806724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.807487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:28.807797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:28.809985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:28.811995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:28.813561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0975a96b-1e48-4de7-a2eb-5657790f96ce None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.833611 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 290/578] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:28.838833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.851759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.852365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.852641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.852910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:28.853578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:28.853984 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:28.855300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:28.863356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 289/579] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:28.864149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:28.954270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abcd42d-2024-428b-99b2-a769203d90a1 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.961399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:28.983384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c4232a-b4df-44e3-980f-84d9ed1010e8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:28.990866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:28.991500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:28.991778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:28.992046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.003637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.005089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.021450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.022098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.022385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.022662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:29.024294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:29.025202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:29.026092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:29.026092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:29.026092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:29.026664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 291/580] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:29.027269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.038045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.039560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:29.054648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:29.055891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:29.057184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:29.057184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:29.057184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:29.057184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 290/581] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:29.057184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.089803 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:29.151360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a90c42-000e-47d2-9d09-1573b8edf51e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.178653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a39958-fd8d-47f1-af62-e9fc3e019e9f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.190709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.191402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.191691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.191966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.203614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.205621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.216384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.217013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.217289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.217556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.220109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:29.220998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:29.222254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:29.223608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:29.223608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:29.223608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:29.223608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 292/582] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:29.223608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.238817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.240323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:29.256730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:29.257763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:29.263553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:29.263553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:29.263553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:29.263553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 291/583] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:29.263553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.338405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:29.346187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4df48d5-b71e-4e46-9264-98ac0ecc7f83 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.373240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af0eaf0-2153-453d-b356-6064a24acb70 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.384864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.385477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.385760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.386032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.386710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:29.387046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.388421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.401636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 293/584] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:29.402529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.415754 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.416359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.416635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.416903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.417689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:29.418509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.419921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.439895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 292/585] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:29.440792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.496884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:29.539997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3cda15-2627-4164-8466-b3ec6b1deddf None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.566087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109d64e-f622-4cff-bbf8-e17a4ef82815 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.581123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.581744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.582017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.582287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.582973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:29.583345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.584650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.597674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 294/586] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:29.598504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.612116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.612960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.613390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.613847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.614510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:29.614847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.616225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.620967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:29.633913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 293/587] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:29.633913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.711771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4889a7a-568c-4888-9e96-2f16062fb84e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.747378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a7c66c-418e-43ba-b645-dfd9459f6e50 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.750481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.751213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.751508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.751999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.757461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:29.759502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.761062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:29.773596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:29.774558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 295/588] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:29.775579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.800643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.801566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.801847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.802117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.812159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.814352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:29.831903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:29.833236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 22:27:29.834478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:29.838194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 294/589] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:29.839129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.872788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:29.900164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12580575-8c83-405c-b1df-8ace8e07f9e0 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.941931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.942573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:29.942987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:29.943260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:29.952004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:29.953431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e52eb-284c-441f-b609-d39d6c99158d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:29.953647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:29.966540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:29.967548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:29.968440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 296/590] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:29.969527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:29.999154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:29.999804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.000084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.000360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.009582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.011487 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:30.036513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:30.037505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 22:27:30.038449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:30.039694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 295/591] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/users/neutron => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:30.040439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:30.087659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb4ad6-c0ca-4f3d-b0c2-50c1fba28ceb None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.088515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:30.124873 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:30.125596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.125937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.126274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.127012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:30.127320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.128750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:30.131634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:30.143088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 297/592] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:30.144003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:30.165214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa4085-8ad0-44e3-9ab7-f07493833503 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.201117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:30.201737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.201994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.202262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.203010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:30.203318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.204744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:30.222688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 296/593] 10.222.0.77 () {56 vars in 1137 bytes} [Mon Aug 29 22:27:30 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:30.223539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:30.245489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81b664c-d6c5-4c1b-8d5f-55af41fb007b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.282117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d9bf24-69dc-430d-a7e6-abae76e23aaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:30.282648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d9bf24-69dc-430d-a7e6-abae76e23aaa None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.282957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d9bf24-69dc-430d-a7e6-abae76e23aaa None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.283230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d9bf24-69dc-430d-a7e6-abae76e23aaa None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.283922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d9bf24-69dc-430d-a7e6-abae76e23aaa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:30.284227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d9bf24-69dc-430d-a7e6-abae76e23aaa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.285711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d9bf24-69dc-430d-a7e6-abae76e23aaa None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:30.298290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 298/594] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:30.299443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f7dce-fccc-4e74-95e9-f9fed6b69980 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.299662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f7dce-fccc-4e74-95e9-f9fed6b69980 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.299938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f7dce-fccc-4e74-95e9-f9fed6b69980 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.302811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 299/595] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:30 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:30.302811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4f7dce-fccc-4e74-95e9-f9fed6b69980 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:30.313986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:30.335755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a798c5-812a-4752-a6e3-a26920c1ce2f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.371793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:30.374123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:30.374737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.375049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.375331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.376068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:30.376379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.377825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:30.396548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 297/596] 10.222.0.77 () {56 vars in 1139 bytes} [Mon Aug 29 22:27:30 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:30.397377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:30.424242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4f7dce-fccc-4e74-95e9-f9fed6b69980 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.459628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc7fc3a-dc5b-4231-9532-59f8238e2edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:30.460291 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7fc3a-dc5b-4231-9532-59f8238e2edd None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.460563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7fc3a-dc5b-4231-9532-59f8238e2edd None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.460837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7fc3a-dc5b-4231-9532-59f8238e2edd None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/c1e61f3604fd4a23ae5f1e7e62ae21ae/roles/6c40acc5170649d8ad758d42c85b91e0` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.475571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc7fc3a-dc5b-4231-9532-59f8238e2edd None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=c1e61f3604fd4a23ae5f1e7e62ae21ae, role_id=6c40acc5170649d8ad758d42c85b91e0)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.477020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc7fc3a-dc5b-4231-9532-59f8238e2edd None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:30.490132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 300/597] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:27:30 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/c1e61f3604fd4a23ae5f1e7e62ae21ae/roles/6c40acc5170649d8ad758d42c85b91e0 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:30.490952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc7fc3a-dc5b-4231-9532-59f8238e2edd None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:30.515768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b663a1-29fd-4382-9bc3-1ac9c9f97db0 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.556194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d04ba55-9f41-4f45-ba8b-959b2b507f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:30.556734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d04ba55-9f41-4f45-ba8b-959b2b507f80 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.557011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d04ba55-9f41-4f45-ba8b-959b2b507f80 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.557278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d04ba55-9f41-4f45-ba8b-959b2b507f80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.570023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d04ba55-9f41-4f45-ba8b-959b2b507f80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.571861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d04ba55-9f41-4f45-ba8b-959b2b507f80 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:30.574933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-8d04ba55-9f41-4f45-ba8b-959b2b507f80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:30.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:30.592450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:30.593549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:30.593549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:30.593549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:30.593841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 298/598] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:30.595263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126f9e77-ae0e-4c32-8f10-228401c24c16 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.595554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126f9e77-ae0e-4c32-8f10-228401c24c16 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.595861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126f9e77-ae0e-4c32-8f10-228401c24c16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.615731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-126f9e77-ae0e-4c32-8f10-228401c24c16 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:30.617064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-126f9e77-ae0e-4c32-8f10-228401c24c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:30.625517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc7fc3a-dc5b-4231-9532-59f8238e2edd None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.633601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:30.671796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e9bf2-1cfc-464a-8227-f4070c5ff984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:30.672406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e9bf2-1cfc-464a-8227-f4070c5ff984 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.672686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e9bf2-1cfc-464a-8227-f4070c5ff984 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.672951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e9bf2-1cfc-464a-8227-f4070c5ff984 None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/57491e4de6c3435e9d818266a910b831/roles/6c40acc5170649d8ad758d42c85b91e0` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.691348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7e9bf2-1cfc-464a-8227-f4070c5ff984 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=57491e4de6c3435e9d818266a910b831, role_id=6c40acc5170649d8ad758d42c85b91e0)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.692902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7e9bf2-1cfc-464a-8227-f4070c5ff984 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:30.707782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 301/599] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:27:30 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/57491e4de6c3435e9d818266a910b831/roles/6c40acc5170649d8ad758d42c85b91e0 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:30.714992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e9bf2-1cfc-464a-8227-f4070c5ff984 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:30.722226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 299/600] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:30.723943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126f9e77-ae0e-4c32-8f10-228401c24c16 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:30.923260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e9bf2-1cfc-464a-8227-f4070c5ff984 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.970356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126f9e77-ae0e-4c32-8f10-228401c24c16 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:30.971356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:30.972453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:30.973848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:30.974303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:30.987895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:30.990444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.003574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:31.013923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:31.015137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:31.016228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:31.016228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:31.016228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:31.017829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 302/601] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:31.018147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.022563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.022563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.022563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.022563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:31.022563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:31.022563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.024512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.037817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 300/602] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:31.044104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.045570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:31.153594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99748f1d-6156-462c-8869-e428a2a7cf78 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:31.207403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.208173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.208560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.209185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.210368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:31.210809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:31.212602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.213816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34fb895-1d23-4d2a-92ce-d8ae5a1b2819 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:31.230060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 303/603] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:31.232629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.256388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.257312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.257819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.258278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.267362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:31.269311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:31.283293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:31.284313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:31.285242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:31.285242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:31.285242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:31.285242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:31.285242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:31.285242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:31.285242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:31.285242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:31.288323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 301/604] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:31.289542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.323190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:31.344314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:31.378391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d02612e-fb26-4eff-abb7-ebc2c24250d9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:31.410383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b955a6f0-bf92-456b-b259-02709457b144 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:31.429200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0615f1-0da1-4814-b8de-5573219cdbba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.430347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0615f1-0da1-4814-b8de-5573219cdbba None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.430811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0615f1-0da1-4814-b8de-5573219cdbba None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.431577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0615f1-0da1-4814-b8de-5573219cdbba None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.447000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0615f1-0da1-4814-b8de-5573219cdbba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:31.450808 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.451651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.451998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.452338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-bd0615f1-0da1-4814-b8de-5573219cdbba None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:31.454086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:31.455230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:31.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:31.459083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 304/605] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:31.460081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0615f1-0da1-4814-b8de-5573219cdbba None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.463348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:31.466204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:31.478832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:31.479971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:31.481272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:31.481272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:31.481272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:31.481272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:31.481272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:31.481272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:31.481272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:31.481272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:31.482785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 302/606] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:31.483755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.608664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0615f1-0da1-4814-b8de-5573219cdbba None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:31.621770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa167797-d1b2-4770-afa1-2f165b2ab9df None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:31.624610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:31.642985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:31.653343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.654136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.654462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.654857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.655740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:31.656154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:31.658114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.665703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.666568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.667064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.667507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.668525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:31.669019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:31.670641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.676276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 305/607] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:31.677390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.681406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 303/608] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:31.682478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.798757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8196dad-bb59-417c-a23c-96c9f55d2222 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:31.811607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc63d03-195b-428d-bc81-74b87b1297d0 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:31.836876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.837517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.837794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.838064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.838780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:31.839171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:31.840989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.853753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1024685a-8101-4aef-bc04-24bcfc3c9d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:31.854713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 306/609] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:31.855055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1024685a-8101-4aef-bc04-24bcfc3c9d6c None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:31.855658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.855769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1024685a-8101-4aef-bc04-24bcfc3c9d6c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:31.856346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1024685a-8101-4aef-bc04-24bcfc3c9d6c None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/7964b4109f9a4ac6a68d90366489557c/roles/6c40acc5170649d8ad758d42c85b91e0` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:31.873921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1024685a-8101-4aef-bc04-24bcfc3c9d6c None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=7964b4109f9a4ac6a68d90366489557c, role_id=6c40acc5170649d8ad758d42c85b91e0)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:31.875867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1024685a-8101-4aef-bc04-24bcfc3c9d6c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:31.893104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 304/610] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:27:31 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/7964b4109f9a4ac6a68d90366489557c/roles/6c40acc5170649d8ad758d42c85b91e0 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:31.894224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1024685a-8101-4aef-bc04-24bcfc3c9d6c None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:31.939482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:31.940979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:32.041392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51356819-b058-4c1b-845e-c78274aa3025 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:32.059077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1024685a-8101-4aef-bc04-24bcfc3c9d6c None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:32.089862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1cc611-4cd1-4bb6-aed0-c225289c4fd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:32.091447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1cc611-4cd1-4bb6-aed0-c225289c4fd6 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:32.091447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1cc611-4cd1-4bb6-aed0-c225289c4fd6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:32.091447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1cc611-4cd1-4bb6-aed0-c225289c4fd6 None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/dea13198ef8e45de85b92e79e3ee71b0/roles/6c40acc5170649d8ad758d42c85b91e0` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:32.111944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:32.113045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:32.113569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:32.114035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:32.133415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:32.136833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1cc611-4cd1-4bb6-aed0-c225289c4fd6 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=dea13198ef8e45de85b92e79e3ee71b0, role_id=6c40acc5170649d8ad758d42c85b91e0)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:32.139555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1cc611-4cd1-4bb6-aed0-c225289c4fd6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:32.142870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:32.155386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 305/611] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:27:31 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/dea13198ef8e45de85b92e79e3ee71b0/roles/6c40acc5170649d8ad758d42c85b91e0 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:32.167251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:32.168367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:32.169579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:32.169579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:32.169579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:32.171718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 307/612] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:32.177664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1cc611-4cd1-4bb6-aed0-c225289c4fd6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:32.262439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:32.304685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1cc611-4cd1-4bb6-aed0-c225289c4fd6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:32.343130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:32.343756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:32.344032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:32.344301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:32.344962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:32.345261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:32.346672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:32.360225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 306/613] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:32.365475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:32.490439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:32.495447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfc962f-04c0-4838-bd3f-d21a04467618 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:32.537151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04218d72-4b08-40bd-bfb5-027f2d925e61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:32.537756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04218d72-4b08-40bd-bfb5-027f2d925e61 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:32.538091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04218d72-4b08-40bd-bfb5-027f2d925e61 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:32.538412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04218d72-4b08-40bd-bfb5-027f2d925e61 None admin] PATH_INFO: `/v3/users/swift` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:32.547859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04218d72-4b08-40bd-bfb5-027f2d925e61 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:32.549428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04218d72-4b08-40bd-bfb5-027f2d925e61 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-04218d72-4b08-40bd-bfb5-027f2d925e61 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:32.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:32.564423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:32.566113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 308/614] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:32.570227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:32.682460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c32155-86b8-4257-870f-c8599319cbb3 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:32.683134 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c32155-86b8-4257-870f-c8599319cbb3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:32.683704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c32155-86b8-4257-870f-c8599319cbb3 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:32.687022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 309/615] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:32.691809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b46dcb-9641-473d-8fda-6d22d2efb6f8 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:32.692461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b46dcb-9641-473d-8fda-6d22d2efb6f8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:32.692954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b46dcb-9641-473d-8fda-6d22d2efb6f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:32.696437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91683507-490a-414e-947a-172aa2b0bcb1 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:32.737045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-05b46dcb-9641-473d-8fda-6d22d2efb6f8 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:32.739742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-05b46dcb-9641-473d-8fda-6d22d2efb6f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:32.760260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:32.779878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:32.780885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:32.781456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:32.782184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:32.783162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:32.783673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:32.787131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:32.825696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 307/616] 10.222.0.77 () {56 vars in 1135 bytes} [Mon Aug 29 22:27:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:32.830777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:32.887960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 310/617] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:33.007381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fdc8c3-29ea-473e-9911-2e2f95513e21 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:33.061932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:33.063420 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:33.064059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:33.064593 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:33.081138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:33.082990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:33.102029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:33.103131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:33.104117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:33.104117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:33.104117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:33.106498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 308/618] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:33.116423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b46dcb-9641-473d-8fda-6d22d2efb6f8 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:33.203608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:33.241000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:33.258253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:33.276282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b46dcb-9641-473d-8fda-6d22d2efb6f8 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:33.319216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96205f48-31d1-4e59-b1af-497455c06c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:33.320296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96205f48-31d1-4e59-b1af-497455c06c45 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:33.320674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96205f48-31d1-4e59-b1af-497455c06c45 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:33.321404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96205f48-31d1-4e59-b1af-497455c06c45 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:33.322317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96205f48-31d1-4e59-b1af-497455c06c45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:33.322912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96205f48-31d1-4e59-b1af-497455c06c45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:33.325311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96205f48-31d1-4e59-b1af-497455c06c45 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:33.335076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5523bf59-5371-4527-94e8-7481a6652006 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:33.340158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 311/619] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:33.341688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d867a5-f6c2-4f0d-9ffb-236da66f9afa None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:33.341976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d867a5-f6c2-4f0d-9ffb-236da66f9afa None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:33.342245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d867a5-f6c2-4f0d-9ffb-236da66f9afa None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:33.347505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 312/620] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:33.348301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d867a5-f6c2-4f0d-9ffb-236da66f9afa None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:33.384016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa7ea26-3f6a-49ec-9817-4b0341c332fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:33.385077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa7ea26-3f6a-49ec-9817-4b0341c332fc None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:33.385863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa7ea26-3f6a-49ec-9817-4b0341c332fc None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:33.386357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa7ea26-3f6a-49ec-9817-4b0341c332fc None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:33.401838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa7ea26-3f6a-49ec-9817-4b0341c332fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2fa7ea26-3f6a-49ec-9817-4b0341c332fc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:33.410033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:33.411263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:33.412392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:33.418143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 309/621] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:33.418143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36171cf2-32fb-4746-bfce-a7e31d6c7360 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:33.418143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36171cf2-32fb-4746-bfce-a7e31d6c7360 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:33.418143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36171cf2-32fb-4746-bfce-a7e31d6c7360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:33.446243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-36171cf2-32fb-4746-bfce-a7e31d6c7360 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:33.447576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-36171cf2-32fb-4746-bfce-a7e31d6c7360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:33.499518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d867a5-f6c2-4f0d-9ffb-236da66f9afa None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:33.543741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 310/622] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:33.545068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36171cf2-32fb-4746-bfce-a7e31d6c7360 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:33.552568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:33.553736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:33.554173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:33.554660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:33.564833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:33.567342 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:33.592599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:33.593537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:33.594418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:33.594418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:33.594418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:33.594418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:33.594418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:33.594418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:33.594418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:33.594418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:33.596965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 313/623] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:33.604942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:33.612068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:33.623722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:33.734001 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ee6a42-33af-47ed-8a51-fe68bcb23153 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:33.762425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36171cf2-32fb-4746-bfce-a7e31d6c7360 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:33.820033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:33.820929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:33.821424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:33.822358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:33.823168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8507608-325a-4215-869c-93a2de07104d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:33.823979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:33.824748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8507608-325a-4215-869c-93a2de07104d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:33.825196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:33.825597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8507608-325a-4215-869c-93a2de07104d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:33.827132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:33.827525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8507608-325a-4215-869c-93a2de07104d None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:33.828369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8507608-325a-4215-869c-93a2de07104d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:33.828785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8507608-325a-4215-869c-93a2de07104d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:33.831711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8507608-325a-4215-869c-93a2de07104d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:33.844821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 311/624] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:33.846472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:33.858874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 314/625] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:33.865314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8507608-325a-4215-869c-93a2de07104d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:33.954500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:33.983390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:33.988268 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6f5b30-d0d3-42d0-a427-546e7d2012ad None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:34.037195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8507608-325a-4215-869c-93a2de07104d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:34.065312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:34.065312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.065312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.065312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.077124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:34.078609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:34.094391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:34.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:34.096655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:34.096655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:34.096655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:34.096655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 312/626] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:34.096655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:34.112227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11495779-52ce-40a9-b441-684dcb18017d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:34.113143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11495779-52ce-40a9-b441-684dcb18017d None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.114035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11495779-52ce-40a9-b441-684dcb18017d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.114472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11495779-52ce-40a9-b441-684dcb18017d None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/03361f4b51d34cf28c4b5451c3b1f881/roles/6c40acc5170649d8ad758d42c85b91e0` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.143452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11495779-52ce-40a9-b441-684dcb18017d None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=03361f4b51d34cf28c4b5451c3b1f881, role_id=6c40acc5170649d8ad758d42c85b91e0)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:34.147016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11495779-52ce-40a9-b441-684dcb18017d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:34.175776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 315/627] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:27:33 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/03361f4b51d34cf28c4b5451c3b1f881/roles/6c40acc5170649d8ad758d42c85b91e0 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:27:34.177435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11495779-52ce-40a9-b441-684dcb18017d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:34.231106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6aefafd-9adb-4c21-9b8e-631991661325 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:34.286516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7be7f1-6ee5-4a8b-929d-25067338c76f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:34.288427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7be7f1-6ee5-4a8b-929d-25067338c76f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.293894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7be7f1-6ee5-4a8b-929d-25067338c76f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.293894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7be7f1-6ee5-4a8b-929d-25067338c76f None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.312053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:34.319172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7be7f1-6ee5-4a8b-929d-25067338c76f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-3d7be7f1-6ee5-4a8b-929d-25067338c76f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:34.326661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:34.328099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:34.329145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:34.330691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 313/628] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:34.331387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac169e-8112-4898-b347-76dc0885c07e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.331684 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac169e-8112-4898-b347-76dc0885c07e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.332090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac169e-8112-4898-b347-76dc0885c07e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.333651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 314/629] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:34.336899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ac169e-8112-4898-b347-76dc0885c07e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:34.353911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11495779-52ce-40a9-b441-684dcb18017d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:34.379108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:34.410186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2816b380-75e3-4953-a29e-d04d20cd683e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:34.410883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2816b380-75e3-4953-a29e-d04d20cd683e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.411170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2816b380-75e3-4953-a29e-d04d20cd683e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.411446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2816b380-75e3-4953-a29e-d04d20cd683e None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.412258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2816b380-75e3-4953-a29e-d04d20cd683e None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:34.412577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2816b380-75e3-4953-a29e-d04d20cd683e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:34.414584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2816b380-75e3-4953-a29e-d04d20cd683e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:34.431970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 316/630] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:34.433255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c5143-9ff1-403b-a9e4-3dff2ab23944 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.433544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c5143-9ff1-403b-a9e4-3dff2ab23944 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.434449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c5143-9ff1-403b-a9e4-3dff2ab23944 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.462321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a14c5143-9ff1-403b-a9e4-3dff2ab23944 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:34.463730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a14c5143-9ff1-403b-a9e4-3dff2ab23944 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:34.475491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ac169e-8112-4898-b347-76dc0885c07e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:34.521636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:34.522316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.523018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.523300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.524258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:34.524573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:34.527591 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:34.540389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 315/631] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:34.541246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:34.551788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 317/632] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:34.552582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14c5143-9ff1-403b-a9e4-3dff2ab23944 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:34.653456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:34.712722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14c5143-9ff1-403b-a9e4-3dff2ab23944 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:34.717882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a3e4ab-a289-4d1f-bab4-2bcbb81aa2d3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:34.725559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:34.754211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb22511-9f9a-4b4c-95f5-616765c11f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:34.754982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb22511-9f9a-4b4c-95f5-616765c11f8c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.755441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb22511-9f9a-4b4c-95f5-616765c11f8c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.755798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb22511-9f9a-4b4c-95f5-616765c11f8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.764746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a37e35-efc6-4cdb-8c67-1a36580a47d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:34.766183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a37e35-efc6-4cdb-8c67-1a36580a47d4 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.766959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a37e35-efc6-4cdb-8c67-1a36580a47d4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.767689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a37e35-efc6-4cdb-8c67-1a36580a47d4 None admin] PATH_INFO: `/v3/users/glance` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.768495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb22511-9f9a-4b4c-95f5-616765c11f8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:34.774775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb22511-9f9a-4b4c-95f5-616765c11f8c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:34.777503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a37e35-efc6-4cdb-8c67-1a36580a47d4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:34.779206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a37e35-efc6-4cdb-8c67-1a36580a47d4 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-bcb22511-9f9a-4b4c-95f5-616765c11f8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:34.787156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:34.789035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:34.790939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:34.790939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:34.790939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:34.790939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 318/633] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:34.790939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac55ab1-fa4d-4ea3-9521-bd1618ac2407 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.790939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac55ab1-fa4d-4ea3-9521-bd1618ac2407 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.790939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac55ab1-fa4d-4ea3-9521-bd1618ac2407 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.792492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 319/634] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:34.798809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac55ab1-fa4d-4ea3-9521-bd1618ac2407 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-61a37e35-efc6-4cdb-8c67-1a36580a47d4 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:34.808218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:34.809954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:34.810964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 316/635] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:34 2022] GET /identity/v3/users/glance => generated 84 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:34.821083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125ede30-9246-4604-ba03-bff182ca7dea None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.821374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125ede30-9246-4604-ba03-bff182ca7dea None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.821648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125ede30-9246-4604-ba03-bff182ca7dea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.842141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-125ede30-9246-4604-ba03-bff182ca7dea None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:34.843437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-125ede30-9246-4604-ba03-bff182ca7dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:34.921143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac55ab1-fa4d-4ea3-9521-bd1618ac2407 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:34.922775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 317/636] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:34.923043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125ede30-9246-4604-ba03-bff182ca7dea None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:34.973382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:34.974317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:34.974891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:34.975310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:34.976130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:34.977193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:34.979242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:34.993091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:35.002673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 320/637] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:35.002673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.027279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:35.056095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125ede30-9246-4604-ba03-bff182ca7dea None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.095176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.096162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.096643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.097110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.098154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:35.098752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:35.100495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:35.107979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc18a8a-e63c-4477-ac2e-96221681d9ef None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.126044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 318/638] 10.222.0.77 () {56 vars in 1137 bytes} [Mon Aug 29 22:27:34 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:35.127393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.144835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a99dd-d214-41ae-93a1-5c2d58c1735e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.145239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a99dd-d214-41ae-93a1-5c2d58c1735e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.145536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a99dd-d214-41ae-93a1-5c2d58c1735e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.145806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a99dd-d214-41ae-93a1-5c2d58c1735e None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.157995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7a99dd-d214-41ae-93a1-5c2d58c1735e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-ce7a99dd-d214-41ae-93a1-5c2d58c1735e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:35.163387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:35.164363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:35.165284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 321/639] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:35.165967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a99dd-d214-41ae-93a1-5c2d58c1735e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.243929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde053a-d640-421f-ac62-a6479968fd65 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.257669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:35.257852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:35.280602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a99dd-d214-41ae-93a1-5c2d58c1735e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.292710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.294016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.294780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.295065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] PATH_INFO: `/v3/users/nova` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.304365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:35.306323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:35.320976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:35.322116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:35.323139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:35.326168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0480d428-c33a-4670-9e3a-b98fe2fc37c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.326814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0480d428-c33a-4670-9e3a-b98fe2fc37c1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.327153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 319/640] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:35.327261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0480d428-c33a-4670-9e3a-b98fe2fc37c1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.327431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0480d428-c33a-4670-9e3a-b98fe2fc37c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.328020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.430243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0480d428-c33a-4670-9e3a-b98fe2fc37c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:35.431787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0480d428-c33a-4670-9e3a-b98fe2fc37c1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:35.443942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a88ef0-5ebc-4fd4-bf97-a1af66cf3437 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.486148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.486786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.487075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.487353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.488027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:35.488333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:35.489727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:35.503421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 320/641] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:35.504122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.537411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:35.540023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:35.571585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 322/642] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:35.571585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0480d428-c33a-4670-9e3a-b98fe2fc37c1 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.630542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7e8077-5317-4bee-941d-9a2363ddcef9 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.673347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03587fbe-35b8-4d9a-a5fb-e93d68588473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.673996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03587fbe-35b8-4d9a-a5fb-e93d68588473 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.674258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03587fbe-35b8-4d9a-a5fb-e93d68588473 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.674519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03587fbe-35b8-4d9a-a5fb-e93d68588473 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.686308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03587fbe-35b8-4d9a-a5fb-e93d68588473 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-03587fbe-35b8-4d9a-a5fb-e93d68588473 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:35.691634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:35.692635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:35.693486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 321/643] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:35.694193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03587fbe-35b8-4d9a-a5fb-e93d68588473 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.708182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0480d428-c33a-4670-9e3a-b98fe2fc37c1 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.751395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.752002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.752288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.752553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.753288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:35.753647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:35.755077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:35.779788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 323/644] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:35.780605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.810884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:35.822483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03587fbe-35b8-4d9a-a5fb-e93d68588473 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.844106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:35.882601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.883278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.883562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.883836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.895997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:35.897524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:35.912817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:35.913827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:35.915368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:35.915368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:35.915368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:35.915368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 322/645] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:35.915368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:35.928645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df605f5-b0d3-46d5-a6ec-889423a0e47d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:35.967636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:35.968269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:35.968610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:35.968887 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:35.979848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:35.981857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:35.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:35.998404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:35.999529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:35.999529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:35.999529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:35.999529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 324/646] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:36.000244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.031830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b930d485-623a-4686-842e-ecd882d1ed1d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.074582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.075361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.075639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.075909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.076577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:36.076878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.078261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.087040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 323/647] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:36.087843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.101015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:36.107482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:36.116921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f248c-7c15-43c3-8250-33b803bc720b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.162435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3f3753-d066-486e-9242-19a488d28aee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.163254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f3753-d066-486e-9242-19a488d28aee None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.163555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f3753-d066-486e-9242-19a488d28aee None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.164031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f3753-d066-486e-9242-19a488d28aee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.206273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcf9624-8259-4dc7-a6be-65974f425afd None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.239303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3f3753-d066-486e-9242-19a488d28aee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.240770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3f3753-d066-486e-9242-19a488d28aee None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.241685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.242294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.242578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.242876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.243521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:36.243819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.245181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.258097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 324/648] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:36.263893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.345522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:36.347763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:36.355179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 325/649] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:36.356619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3f3753-d066-486e-9242-19a488d28aee None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.387842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261bdf72-cd16-49c5-9539-577c3462a700 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.424070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.424678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.424965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.425233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.425892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:36.426189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.427618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.439366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 325/650] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:36.440205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.475785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3f3753-d066-486e-9242-19a488d28aee None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.511376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96073257-ad48-472b-a515-6fc59b1b838d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.512037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96073257-ad48-472b-a515-6fc59b1b838d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.512319 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96073257-ad48-472b-a515-6fc59b1b838d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.512591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96073257-ad48-472b-a515-6fc59b1b838d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.522785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96073257-ad48-472b-a515-6fc59b1b838d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.524246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96073257-ad48-472b-a515-6fc59b1b838d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-96073257-ad48-472b-a515-6fc59b1b838d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:36.540566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:36.541561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 326/651] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7d1cc-9853-4d23-99b0-7dc221bb4a2a None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7d1cc-9853-4d23-99b0-7dc221bb4a2a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7d1cc-9853-4d23-99b0-7dc221bb4a2a None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 327/652] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:36.548730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c7d1cc-9853-4d23-99b0-7dc221bb4a2a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.550916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426af6ac-4ae9-4bfb-b8b5-716181c0186c None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.577703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:36.587365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.588412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.588693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.588966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.594493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:36.598076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.599477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:36.611717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:36.612688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 326/653] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:36.613645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.669247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c7d1cc-9853-4d23-99b0-7dc221bb4a2a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.704351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.704978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.705265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.705544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.716770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.718246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.726512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a894193-fb88-4d6b-ba44-891846d6227a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:36.742167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:36.743197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:36.744094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:36.744094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:36.744094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:36.744417 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 328/654] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:36.745496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.763833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399b9431-e92b-4712-aa54-31ef690dde9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.764466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b9431-e92b-4712-aa54-31ef690dde9a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.764747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b9431-e92b-4712-aa54-31ef690dde9a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.765018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399b9431-e92b-4712-aa54-31ef690dde9a None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.771709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399b9431-e92b-4712-aa54-31ef690dde9a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.773150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399b9431-e92b-4712-aa54-31ef690dde9a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-399b9431-e92b-4712-aa54-31ef690dde9a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:36.785643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:36.791539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 327/655] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4924b5b4-305b-43ff-bc1f-b41effcde577 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.792483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4924b5b4-305b-43ff-bc1f-b41effcde577 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.793083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4924b5b4-305b-43ff-bc1f-b41effcde577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.806326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-4924b5b4-305b-43ff-bc1f-b41effcde577 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:36.807562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4924b5b4-305b-43ff-bc1f-b41effcde577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:36.848439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:36.852628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:36.862070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c24034-1bf7-4f67-a7ee-20d96e50f89b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:36.879770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 328/656] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:36.891010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4924b5b4-305b-43ff-bc1f-b41effcde577 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:36.908397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:36.909095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:36.909387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:36.909664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:36.910321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:36.910634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:36.912070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:36.928386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 329/657] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:36.928386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.017084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4924b5b4-305b-43ff-bc1f-b41effcde577 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.043597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02da876-0330-4cd6-b662-b55d153822a6 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.072329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.072938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.073221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.073499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.074379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:37.075425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.076876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.080877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.081492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.081771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.082039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.082684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:37.083019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.084432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.099778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 329/658] 10.222.0.77 () {56 vars in 1137 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:37.099778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.100946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 330/659] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:37.101869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.120962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:37.138410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:37.206940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5cedb8-190f-4614-95d1-a2a2c20e6f94 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.207155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830edcae-1f2a-4bae-8009-8f129cd4463d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.241547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.242047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.242192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.242448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.242653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.242776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.242967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.243240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.243939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:37.244244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.245696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.251232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.252680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.257982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 330/660] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:37.258821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:37.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:37.265808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 331/661] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/users/neutron => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:37.267027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.350936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:37.356786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:37.364170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb54b567-7d27-4c4b-8e42-63e9834d7bac None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.376081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519ab9e3-2e25-4500-a00a-02fd745254f3 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.400655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebdd815-1751-4db2-b0d2-012bafd50524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.401265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebdd815-1751-4db2-b0d2-012bafd50524 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.401547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebdd815-1751-4db2-b0d2-012bafd50524 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.401820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebdd815-1751-4db2-b0d2-012bafd50524 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.411930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.412563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.412847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.413125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.420147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.421596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:37.432876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:37.434479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 332/662] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:37.435629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.478821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebdd815-1751-4db2-b0d2-012bafd50524 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.480300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebdd815-1751-4db2-b0d2-012bafd50524 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.542885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f57dec2-0b11-430c-ad21-1f82a49c2d01 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.573808 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:37.577609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4031343-c335-4a64-8acc-76f21e727ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.578233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4031343-c335-4a64-8acc-76f21e727ebe None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.578518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4031343-c335-4a64-8acc-76f21e727ebe None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.578818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4031343-c335-4a64-8acc-76f21e727ebe None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.581493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:37.582167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 331/663] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:37.582979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebdd815-1751-4db2-b0d2-012bafd50524 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.592545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4031343-c335-4a64-8acc-76f21e727ebe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-f4031343-c335-4a64-8acc-76f21e727ebe None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:37.598060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:37.599447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 333/664] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:37.600312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4031343-c335-4a64-8acc-76f21e727ebe None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.687009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebdd815-1751-4db2-b0d2-012bafd50524 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.700227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4031343-c335-4a64-8acc-76f21e727ebe None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.720865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.721469 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.721746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.722018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.722752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=c1e61f3604fd4a23ae5f1e7e62ae21ae) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:37.723082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.724526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.730585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 332/665] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/role_assignments?user.id=c1e61f3604fd4a23ae5f1e7e62ae21ae&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:37.732087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.734977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.735679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.736038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.736388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.737231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:37.737614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.739229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.770407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 334/666] 10.222.0.77 () {56 vars in 1139 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:37.771268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.810910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:37.845198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:37.875416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1681c94c-2321-4055-8f6c-ec8610150b1e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.904768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29fdb89-87e9-4fa2-b637-ac3eaf1c5ff9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:37.916968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.917625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.917902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.918170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.920574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:37.920817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.922288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.937454 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 333/667] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:37.938314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:37.953354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:37.954168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:37.954541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:37.954936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:37.967711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:37.969364 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:37.989169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:37.990395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:37.991750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:37.991750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:37.991750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:37.992116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 335/668] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:37.993073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.052772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2451583d-cf0f-407e-8dc5-89ba2e1c7020 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.070807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:38.088668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.089515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.089799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.090079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.091452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:38.091760 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.093139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:38.106780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 334/669] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:38.106780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.120129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b09eff4-0427-4695-ac52-52111aaf4b03 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.127539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:38.158038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223a7422-01b1-4d3c-b648-2f30ee6641c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.158916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a7422-01b1-4d3c-b648-2f30ee6641c6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.159268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a7422-01b1-4d3c-b648-2f30ee6641c6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.159613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a7422-01b1-4d3c-b648-2f30ee6641c6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.220836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37afc5bb-4fc1-4046-95cc-e70a773e5af0 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.242976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223a7422-01b1-4d3c-b648-2f30ee6641c6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.244767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223a7422-01b1-4d3c-b648-2f30ee6641c6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:38.255101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.255713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.255986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.256259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.256986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=57491e4de6c3435e9d818266a910b831) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:38.257292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.258717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:38.265167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 335/670] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:38 2022] GET /identity/v3/role_assignments?user.id=57491e4de6c3435e9d818266a910b831&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:38.265959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.335236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:38.393498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 336/671] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:38.394467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223a7422-01b1-4d3c-b648-2f30ee6641c6 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.406069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:38.412476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec1e19-174b-4b6c-845a-e724340d03ed None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.453666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.454585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.455509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.456026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.456841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:38.457152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.458809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:38.476654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 336/672] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:38.476654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.535959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223a7422-01b1-4d3c-b648-2f30ee6641c6 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.583734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.584743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.585240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.585438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.597418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.599036 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:38.605335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e13db0-baf2-431c-8895-01ec0526f865 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.608956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:38.615121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:38.616100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:38.617020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:38.617020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:38.617020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:38.617020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 337/673] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:38.617020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.657830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.658913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.659402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.659876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.670886 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.672678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:38.689638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:38.690639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:38.691625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:38.691625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:38.691625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:38.692919 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:38.694011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 337/674] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:38.695046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.745120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238c9668-bdcf-43ec-9933-cddc4f90d80d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.787033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.787033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.787033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.787033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.796222 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.797737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:38.808103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1b54b5-85b9-48f9-8f58-de1eac852670 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:38.811241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:38.812136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:38.813037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 338/675] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:38.813979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.852163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.853144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.853648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.854134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.855051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:38.855571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.857275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:38.870186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 338/676] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:38.871365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:38.898275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:38.942029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1e8c4d-695e-443b-ae0f-4f88d0bfe136 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.979176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:38.991743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:38.992941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:38.993434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:38.993895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:38.994707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a478a0fc-53e8-4486-ab1f-b132f15aeaa1 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:38.994936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:38.995584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:38.998071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:39.017708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 339/677] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:39.018920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:39.040931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:39.041486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:39.041771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:39.042081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:39.042819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:39.043113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:39.045284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:39.060436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 339/678] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:39.061499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:39.148852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:39.161473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a0740-ad3b-49c9-9fe5-1d89b0d3d993 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:39.178652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52faf43-155a-4101-90e8-656300b55676 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:39.201264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:39.201971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:39.202253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:39.202523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] PATH_INFO: `/v3/users/swift` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:39.210404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:39.212031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:39.225311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:39.226311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:39.226839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:39.227368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:39.232124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:39.233064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:39.234034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:39.239214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:39.240928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:39.241860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 340/679] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:39.243652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:39.256455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:39.257408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:39.258688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:39.258688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:39.258688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:39.258688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:39.258688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:39.258688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:39.258688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:39.258688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:39.262799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 340/680] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:39.264375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:39.312156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:39.393707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5c7402-7417-4e0a-8420-9da038ae6aed None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:39.424379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c64472-304f-4ba9-aeed-88a7e1ed1995 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:39.445237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:39.446594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:39.447765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:39.448885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:39.451008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=7964b4109f9a4ac6a68d90366489557c) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:39.452078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:39.455280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:39.471627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 341/681] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:39 2022] GET /identity/v3/role_assignments?user.id=7964b4109f9a4ac6a68d90366489557c&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 608 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:39.474272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:39.479078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:39.480024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:39.480626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:39.481660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:39.483208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:39.483521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:39.486242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:39.491991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:39.526924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 341/682] 10.222.0.77 () {56 vars in 1135 bytes} [Mon Aug 29 22:27:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:39.535997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:39.649385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c87fdb-6fd2-4975-89ea-0924e1faccbd None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:39.653675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:39.721676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:39.722307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:39.722589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:39.722949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:39.723676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:39.724042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:39.725530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:39.740908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 342/683] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:39.748720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:39.787313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fee1db3-4adb-4f74-a01f-bb6e9d0f1602 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:39.873834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92dae58-bc18-4b29-9790-17bc8f7d474c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:39.875199 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92dae58-bc18-4b29-9790-17bc8f7d474c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:39.875891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92dae58-bc18-4b29-9790-17bc8f7d474c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:39.876401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92dae58-bc18-4b29-9790-17bc8f7d474c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:39.898644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b737bd24-141c-436a-b149-b840657b4b3f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:39.938876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:39.939496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:39.939772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:39.940037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:39.940774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=dea13198ef8e45de85b92e79e3ee71b0) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:39.941078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:39.942518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:39.949256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 343/684] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:39 2022] GET /identity/v3/role_assignments?user.id=dea13198ef8e45de85b92e79e3ee71b0&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:39.960405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:39.972623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92dae58-bc18-4b29-9790-17bc8f7d474c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:39.974141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92dae58-bc18-4b29-9790-17bc8f7d474c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:40.107531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 342/685] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 573 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:40.113376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:40.167495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:40.265463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb08a78-ed4d-44fc-ba4a-b792d4c41288 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:40.313949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9cad42-80c5-4ee8-bdb6-d9f162b08a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:40.314596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9cad42-80c5-4ee8-bdb6-d9f162b08a17 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:40.314904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9cad42-80c5-4ee8-bdb6-d9f162b08a17 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:40.315176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9cad42-80c5-4ee8-bdb6-d9f162b08a17 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:40.340861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9cad42-80c5-4ee8-bdb6-d9f162b08a17 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:40.343649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9cad42-80c5-4ee8-bdb6-d9f162b08a17 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-1d9cad42-80c5-4ee8-bdb6-d9f162b08a17 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:40.372429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:40.374243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:40.375922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:40.375922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:40.375922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:40.375922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 344/686] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:40.382723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92dae58-bc18-4b29-9790-17bc8f7d474c None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:40.504359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:40.516819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92dae58-bc18-4b29-9790-17bc8f7d474c None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:40.524767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072009c0-2362-4cd8-ac52-bf8837b4d6d6 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:40.525251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072009c0-2362-4cd8-ac52-bf8837b4d6d6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:40.525660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072009c0-2362-4cd8-ac52-bf8837b4d6d6 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:40.528325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 345/687] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:40.537147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bc7315-3dbe-4313-ad94-5fd2b9306dec None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:40.537481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bc7315-3dbe-4313-ad94-5fd2b9306dec None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:40.537698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bc7315-3dbe-4313-ad94-5fd2b9306dec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:40.572890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-21bc7315-3dbe-4313-ad94-5fd2b9306dec None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:40.574220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-21bc7315-3dbe-4313-ad94-5fd2b9306dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:40.577327 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:40.578509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:40.579319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:40.580102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:40.581639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:40.582486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:40.585618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:40.623318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 343/688] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:40.632581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:40.766502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 346/689] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:40.893074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2b351-c11a-4adf-b78c-cad4789e4db6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:40.920168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349d699-4424-49bf-bf44-0b913b2b1f07 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:40.920481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349d699-4424-49bf-bf44-0b913b2b1f07 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:40.920723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b349d699-4424-49bf-bf44-0b913b2b1f07 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:40.922470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 347/690] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:27:40.927571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872957eb-6c11-4918-93ef-d5679c9c2009 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:40.927860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872957eb-6c11-4918-93ef-d5679c9c2009 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:40.928203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872957eb-6c11-4918-93ef-d5679c9c2009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:40.948694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-872957eb-6c11-4918-93ef-d5679c9c2009 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:40.950679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:40.950917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-872957eb-6c11-4918-93ef-d5679c9c2009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:40.951748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:40.952226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:40.952693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:40.955185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:40.970908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:40.972386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:40.988963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:40.989947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:40.990927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 344/691] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 359 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:41.009718 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:41.044879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:41.137324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 348/692] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:41.148768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef95d79-03ca-4ff7-8e36-83804442a9b1 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:41.168201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872957eb-6c11-4918-93ef-d5679c9c2009 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:41.190985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:41.191596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:41.191876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:41.192142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:41.192843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:41.193150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:41.195156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:41.214814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 345/693] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:41.217273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:41.320309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:41.349801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96d32c4-8271-4069-afbc-5415b9b4603a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:41.391132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:41.391966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:41.392244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:41.392515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:41.393251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=03361f4b51d34cf28c4b5451c3b1f881) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:41.393556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:41.395031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:41.411372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 346/694] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:41 2022] GET /identity/v3/role_assignments?user.id=03361f4b51d34cf28c4b5451c3b1f881&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 608 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:41.427127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872957eb-6c11-4918-93ef-d5679c9c2009 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:41.507761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:41.508856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:41.509353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:41.509843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:41.527395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:41.530840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:41.534144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:41.564095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:41.565116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:41.566022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:41.566022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:41.566022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:41.570756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 349/695] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 399 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:41.572719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:41.626626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:41.766391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70556a-587b-4ce9-bed4-17f199e69aaa None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:41.813360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f41258b-6a6d-4c27-b665-48287a48d0b2 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:41.825909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:41.826947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:41.827425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:41.827927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:41.828862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:41.829361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:41.830981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:41.839801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:41.846535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 347/696] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:41.852372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:41.889285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f17e2ce-97e2-4066-bfd1-0b4cd06c0afa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:41.890724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f17e2ce-97e2-4066-bfd1-0b4cd06c0afa None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:41.891264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f17e2ce-97e2-4066-bfd1-0b4cd06c0afa None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:41.891879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f17e2ce-97e2-4066-bfd1-0b4cd06c0afa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:41.918196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f17e2ce-97e2-4066-bfd1-0b4cd06c0afa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-4f17e2ce-97e2-4066-bfd1-0b4cd06c0afa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:41.925200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:41.927018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:41.929221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 350/697] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:41.933669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f17e2ce-97e2-4066-bfd1-0b4cd06c0afa None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:42.026293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea080c-fa1e-42bd-b2af-b7e7cf218057 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:42.054750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:42.092263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2826148e-efd0-4dd8-b643-50eea9370764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:42.095326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2826148e-efd0-4dd8-b643-50eea9370764 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:42.096196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2826148e-efd0-4dd8-b643-50eea9370764 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:42.096979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2826148e-efd0-4dd8-b643-50eea9370764 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:42.098535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2826148e-efd0-4dd8-b643-50eea9370764 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:42.102655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2826148e-efd0-4dd8-b643-50eea9370764 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:42.150923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 348/698] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:27:41 2022] POST /identity/v3/users => generated 254 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:42.154345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f17e2ce-97e2-4066-bfd1-0b4cd06c0afa None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:42.237799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f599b0-349c-4a04-930e-d45a641278f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:42.238837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f599b0-349c-4a04-930e-d45a641278f3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:42.239141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f599b0-349c-4a04-930e-d45a641278f3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:42.239399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f599b0-349c-4a04-930e-d45a641278f3 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:42.240399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f599b0-349c-4a04-930e-d45a641278f3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:42.240711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f599b0-349c-4a04-930e-d45a641278f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:42.242264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f599b0-349c-4a04-930e-d45a641278f3 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:42.258996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 351/699] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:42.268304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2826148e-efd0-4dd8-b643-50eea9370764 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:42.432230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:42.461684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e5dd6-97bc-4c5f-af2c-999ada2f5d19 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:42.462160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e5dd6-97bc-4c5f-af2c-999ada2f5d19 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:42.462356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e5dd6-97bc-4c5f-af2c-999ada2f5d19 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:42.463928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 352/700] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:42.472823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3940390-25db-429d-8fed-09e26006ea10 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:42.473418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3940390-25db-429d-8fed-09e26006ea10 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:42.473732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3940390-25db-429d-8fed-09e26006ea10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:42.501627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f3940390-25db-429d-8fed-09e26006ea10 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:42.504248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f3940390-25db-429d-8fed-09e26006ea10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:42.515480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2826148e-efd0-4dd8-b643-50eea9370764 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:42.572493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:42.573214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:42.573543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:42.573867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:42.592666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:42.594298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:42.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:42.628714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:42.630281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:42.630281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:42.630281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:42.633633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 349/701] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 366 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:42.636629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 353/702] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 29 22:27:42.642871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:42.816545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8706e5-92a8-48c0-9227-8fad9f5f0e5d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:42.827482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:42.866176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:42.866893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:42.867178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:42.867497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:42.868224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:42.868540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:42.870254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:42.884858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 350/703] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:42.893300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3940390-25db-429d-8fed-09e26006ea10 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:42.989148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:43.040468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:43.113815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3940390-25db-429d-8fed-09e26006ea10 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:43.171569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373cf15e-a1a5-4166-9321-9e22822ed468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:43.172370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373cf15e-a1a5-4166-9321-9e22822ed468 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:43.172672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373cf15e-a1a5-4166-9321-9e22822ed468 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:43.173095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373cf15e-a1a5-4166-9321-9e22822ed468 None admin] PATH_INFO: `/v3/users/nova` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:43.182477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373cf15e-a1a5-4166-9321-9e22822ed468 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:43.192237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373cf15e-a1a5-4166-9321-9e22822ed468 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-373cf15e-a1a5-4166-9321-9e22822ed468 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:43.221752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:43.222778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:43.223720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 354/704] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:27:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:43.224859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67341efb-73fb-44c0-bc44-b2860d994f91 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:43.225074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67341efb-73fb-44c0-bc44-b2860d994f91 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:43.225355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67341efb-73fb-44c0-bc44-b2860d994f91 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:43.225539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0a60c3-5ee9-44c6-979d-cc1e83c01b95 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:43.226886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 355/705] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:43.234047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4713c-761a-49af-876d-de61da48c57e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:43.234885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4713c-761a-49af-876d-de61da48c57e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:43.235435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4713c-761a-49af-876d-de61da48c57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:43.256989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-46c4713c-761a-49af-876d-de61da48c57e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:43.258992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-46c4713c-761a-49af-876d-de61da48c57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:43.267826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:43.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:43.268756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:43.269038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:43.269804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:43.271542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:43.289099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:43.290423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:43.291997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 351/706] 10.222.0.77 () {56 vars in 1129 bytes} [Mon Aug 29 22:27:43 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:43.293282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:43.351425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:43.387455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 356/707] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:43.389379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c4713c-761a-49af-876d-de61da48c57e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:43.436816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a426f9a-d5da-42b1-bcde-a4684ede6858 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:43.482559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:43.508615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:43.509703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:43.509703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:43.510014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:43.510701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:43.511034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:43.513432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:43.554761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 352/708] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:27:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:43.564208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:43.646400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c4713c-761a-49af-876d-de61da48c57e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:43.751875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:43.753369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:43.754098 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:43.754860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:43.756208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:43.757062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:43.759859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:43.764089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:43.773104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 357/709] 10.222.0.77 () {56 vars in 1124 bytes} [Mon Aug 29 22:27:43 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 385 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:43.780478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:43.786604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e952bcd8-302d-4cc2-9f91-ecbcdd83f232 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:43.842246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acebb3a5-7c24-494e-aefd-79242c09031b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:43.844914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acebb3a5-7c24-494e-aefd-79242c09031b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:43.845789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acebb3a5-7c24-494e-aefd-79242c09031b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:43.847824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acebb3a5-7c24-494e-aefd-79242c09031b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:43.965267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:43.974995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f0531-31ac-46c2-87e3-a9dc8d94ceda None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:43.982204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acebb3a5-7c24-494e-aefd-79242c09031b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:43.985246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acebb3a5-7c24-494e-aefd-79242c09031b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:44.047745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:44.047745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:44.048350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:44.048920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:44.050150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:44.052340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:44.054986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:44.067913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 358/710] 10.222.0.77 () {56 vars in 1124 bytes} [Mon Aug 29 22:27:43 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:44.072426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:44.211124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:44.252120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 353/711] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 680 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 29 22:27:44.253577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acebb3a5-7c24-494e-aefd-79242c09031b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:44.348776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b495227-c433-454f-b6c7-836247e5c3aa None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:44.398817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:44.399487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:44.399853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:44.400194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] PATH_INFO: `/v3/services/network` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:44.400802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:44.402354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:44.411655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acebb3a5-7c24-494e-aefd-79242c09031b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:44.420500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:44.421643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:44.423243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 359/712] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:44 2022] GET /identity/v3/services/network => generated 88 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:44.430824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:44.472093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:44.472742 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:44.473021 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:44.473472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:44.487304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:44.488902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:44.509671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:44.510788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:44.511722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:44.511722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:44.511722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:44.511722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 354/713] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:44.519522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:44.522957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:44.643131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:44.711823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2828d78-1c06-4f99-8b89-3225e2db1720 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:44.750679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c0f78-9f64-4369-a82e-af1ce157f581 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:44.767989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a249db14-a7fe-4e36-a6ea-f7423977e9d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:44.769041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a249db14-a7fe-4e36-a6ea-f7423977e9d4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:44.769525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a249db14-a7fe-4e36-a6ea-f7423977e9d4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:44.770014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a249db14-a7fe-4e36-a6ea-f7423977e9d4 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:44.771459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a249db14-a7fe-4e36-a6ea-f7423977e9d4 None admin] RBAC: Adding query filter params (name=network) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:44.772513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a249db14-a7fe-4e36-a6ea-f7423977e9d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:44.773921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a249db14-a7fe-4e36-a6ea-f7423977e9d4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:44.780693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 360/714] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:27:44 2022] GET /identity/v3/services?name=network => generated 124 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:44.782168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2165081c-3126-4acc-9080-5cd9f091282c None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:44.782413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2165081c-3126-4acc-9080-5cd9f091282c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:44.783180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2165081c-3126-4acc-9080-5cd9f091282c None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:44.797621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 361/715] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:44 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:44.797621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2165081c-3126-4acc-9080-5cd9f091282c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:44.824569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06c0855-cd36-4a3a-b05c-1e4e6a1320b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:44.825492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c0855-cd36-4a3a-b05c-1e4e6a1320b1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:44.825782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c0855-cd36-4a3a-b05c-1e4e6a1320b1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:44.826065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c0855-cd36-4a3a-b05c-1e4e6a1320b1 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:44.827205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06c0855-cd36-4a3a-b05c-1e4e6a1320b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:44.827528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06c0855-cd36-4a3a-b05c-1e4e6a1320b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:44.829048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06c0855-cd36-4a3a-b05c-1e4e6a1320b1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:44.851597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 355/716] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:44.853958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced6b69-98e7-4afc-b4fa-b6ab5e6c386d None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:44.854477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced6b69-98e7-4afc-b4fa-b6ab5e6c386d None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:44.854993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced6b69-98e7-4afc-b4fa-b6ab5e6c386d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:44.882097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-fced6b69-98e7-4afc-b4fa-b6ab5e6c386d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:44.884295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fced6b69-98e7-4afc-b4fa-b6ab5e6c386d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:44.928504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:44.943507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2165081c-3126-4acc-9080-5cd9f091282c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:44.976231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 356/717] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:44.977537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fced6b69-98e7-4afc-b4fa-b6ab5e6c386d None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:44.982713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:44.983500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:44.983902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:44.984298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:44.985079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] RBAC: Adding query filter params (type=network) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:44.985575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:44.987450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:44.993148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 362/718] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:27:44 2022] GET /identity/v3/services?type=network => generated 124 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:45.025926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:45.136764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fced6b69-98e7-4afc-b4fa-b6ab5e6c386d None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:45.181005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec4c056-cd0c-4b14-9dcb-580271162f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:45.181965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec4c056-cd0c-4b14-9dcb-580271162f97 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:45.182405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec4c056-cd0c-4b14-9dcb-580271162f97 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:45.182863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec4c056-cd0c-4b14-9dcb-580271162f97 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:45.191195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec4c056-cd0c-4b14-9dcb-580271162f97 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:45.192918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec4c056-cd0c-4b14-9dcb-580271162f97 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-eec4c056-cd0c-4b14-9dcb-580271162f97 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:45.209696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:45.210873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:45.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:45.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:45.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:45.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:45.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:45.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:45.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:45.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:45.214856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 357/719] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:45.224269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:45.327908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4049d5-6c59-46e1-ac0d-8c5cd024904c None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:45.328213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4049d5-6c59-46e1-ac0d-8c5cd024904c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:45.328380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4049d5-6c59-46e1-ac0d-8c5cd024904c None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:45.329949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 358/720] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:45.337407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44babef-2ad1-44e1-9548-9ca3a96d45c9 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:45.337894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44babef-2ad1-44e1-9548-9ca3a96d45c9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:45.338383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44babef-2ad1-44e1-9548-9ca3a96d45c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:45.360285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf0275e-e972-43c6-a929-3ebc08502722 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:45.404096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b44babef-2ad1-44e1-9548-9ca3a96d45c9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:45.406363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b44babef-2ad1-44e1-9548-9ca3a96d45c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:45.408997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:45.410549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:45.411461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:45.412297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:45.413848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:45.414650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:45.419162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:45.435535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 363/721] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:45.447864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:45.485406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:45.535768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 359/722] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:45.537312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44babef-2ad1-44e1-9548-9ca3a96d45c9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:45.563558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:45.667048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44babef-2ad1-44e1-9548-9ca3a96d45c9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:45.696121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c1c389-b8df-49ee-b674-732ffdf6ab49 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:45.710355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43223bf5-aaba-4eca-a9c9-ae9ce9364b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:45.712028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43223bf5-aaba-4eca-a9c9-ae9ce9364b58 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:45.712539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43223bf5-aaba-4eca-a9c9-ae9ce9364b58 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:45.713016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43223bf5-aaba-4eca-a9c9-ae9ce9364b58 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:45.728657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43223bf5-aaba-4eca-a9c9-ae9ce9364b58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-43223bf5-aaba-4eca-a9c9-ae9ce9364b58 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:45.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:45.736337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:45.737207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:45.740239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 360/723] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:45.747370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43223bf5-aaba-4eca-a9c9-ae9ce9364b58 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:45.792129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:45.792747 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:45.793026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:45.793299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:45.794824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=57491e4de6c3435e9d818266a910b831) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:45.795267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:45.797924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:45.808384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 364/724] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:45 2022] GET /identity/v3/role_assignments?user.id=57491e4de6c3435e9d818266a910b831&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=38171c89968943bd92d9e9758db65615 => generated 259 bytes in 360 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:45.901515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:45.940097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43223bf5-aaba-4eca-a9c9-ae9ce9364b58 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:45.991734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:45.992505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:45.993132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:45.993415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:45.993691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:45.994378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:45.994688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:45.996153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:46.007262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 361/725] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:46.015816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:46.097883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:46.124531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763577a-8539-4923-ac24-c425893b31d9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:46.133182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239e5858-809e-4b7d-a08f-ebec074a5fb9 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:46.172374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f597a3d-1a86-42d7-a3ae-16ca518aa7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:46.173310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:46.174184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:46.175057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:46.175509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:46.175975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f597a3d-1a86-42d7-a3ae-16ca518aa7f9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:46.177079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f597a3d-1a86-42d7-a3ae-16ca518aa7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:46.177559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f597a3d-1a86-42d7-a3ae-16ca518aa7f9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:46.197737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:46.201703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:46.203147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f597a3d-1a86-42d7-a3ae-16ca518aa7f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-7f597a3d-1a86-42d7-a3ae-16ca518aa7f9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:46.208960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:46.209916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:46.210810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:46.220161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 365/726] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:46.220683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:46.226639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f597a3d-1a86-42d7-a3ae-16ca518aa7f9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:46.248801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:46.249935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:46.255632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:46.255632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:46.255632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:46.255632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 362/727] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:46.274771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:46.512339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f597a3d-1a86-42d7-a3ae-16ca518aa7f9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:46.579411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965d57b-9232-4c3c-9708-a4bf49ec143f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:46.596402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:46.597038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:46.597320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:46.597593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:46.598278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] RBAC: Adding query filter params (name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:46.598585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:46.600210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:46.604983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:46.618034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 366/728] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 391 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:46.624735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:46.662243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:46.662885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:46.663171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:46.663422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:46.665760 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:46.666081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:46.667513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:46.695661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 363/729] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 427 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:46.702363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:46.833266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:46.861017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaad04f-0b55-4221-bfc5-465b0da36962 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:46.914123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8658d635-79f6-4a87-8e3b-4a15963ac44e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:46.935671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:46.936684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:46.937407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:46.938589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:46.959494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:46.962402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:46.990180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:46.991130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:46.991587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:46.992033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] PATH_INFO: `/v3/users/swift` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:46.994450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:46.995764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:46.996657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:46.996657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:46.996657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:47.003206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:47.004728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:47.005767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 367/730] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 381 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:47.012581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:47.032569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:47.033855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:47.035657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:47.038935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 364/731] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:46 2022] GET /identity/v3/users/swift => generated 83 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:47.044723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:47.124318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:47.212752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89fe82a-5e53-4f73-a4a2-d6b9a3dbb941 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:47.228857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2141cfe0-7598-4236-97df-4ac8c77c1f2a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:47.235036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:47.268124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:47.269140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.269635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.270130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.270429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253d7df-d55c-42fc-915b-ef406faed0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:47.271271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253d7df-d55c-42fc-915b-ef406faed0c0 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.271433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253d7df-d55c-42fc-915b-ef406faed0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.271689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253d7df-d55c-42fc-915b-ef406faed0c0 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.272316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3253d7df-d55c-42fc-915b-ef406faed0c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:47.272627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3253d7df-d55c-42fc-915b-ef406faed0c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:47.272810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:47.273320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:47.274027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3253d7df-d55c-42fc-915b-ef406faed0c0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:47.275310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:47.299872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 368/732] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:47.299872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c4ecd-0977-4c0a-bc04-ff872e2f947a None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.299872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c4ecd-0977-4c0a-bc04-ff872e2f947a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.299872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c4ecd-0977-4c0a-bc04-ff872e2f947a None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.299872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 369/733] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:47.302914 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e2364-113b-41c9-8060-b909a726ad5f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.303205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e2364-113b-41c9-8060-b909a726ad5f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.303482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e2364-113b-41c9-8060-b909a726ad5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.311614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 365/734] 10.222.0.77 () {56 vars in 1135 bytes} [Mon Aug 29 22:27:47 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:47.312630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:47.332974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-475e2364-113b-41c9-8060-b909a726ad5f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:47.334226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-475e2364-113b-41c9-8060-b909a726ad5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:47.419378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 370/735] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:47.419378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475e2364-113b-41c9-8060-b909a726ad5f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:47.458434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1f517d-e991-4df6-b9a6-438a47459ecb None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:47.469122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:47.519784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de3ad8-3b73-4cef-ace7-7a69fece790c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:47.520775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de3ad8-3b73-4cef-ace7-7a69fece790c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.521291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de3ad8-3b73-4cef-ace7-7a69fece790c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.521774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de3ad8-3b73-4cef-ace7-7a69fece790c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.536353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475e2364-113b-41c9-8060-b909a726ad5f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:47.538197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8de3ad8-3b73-4cef-ace7-7a69fece790c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:47.541998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8de3ad8-3b73-4cef-ace7-7a69fece790c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-f8de3ad8-3b73-4cef-ace7-7a69fece790c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:47.570608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:47.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:27:47.572679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:47.578034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 366/736] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:27:47 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:47.578034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a832a-4951-4870-9f55-e6396f883756 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.578034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a832a-4951-4870-9f55-e6396f883756 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.578034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a832a-4951-4870-9f55-e6396f883756 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.586381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 367/737] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:47.586545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25d99f7-48bf-435d-90e8-e8025e4e7cd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:47.587617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25d99f7-48bf-435d-90e8-e8025e4e7cd0 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.587729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a832a-4951-4870-9f55-e6396f883756 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:47.588122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25d99f7-48bf-435d-90e8-e8025e4e7cd0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.588620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25d99f7-48bf-435d-90e8-e8025e4e7cd0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.646203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:47.696316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25d99f7-48bf-435d-90e8-e8025e4e7cd0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:47.697714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25d99f7-48bf-435d-90e8-e8025e4e7cd0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:47.742123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:47.747976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a832a-4951-4870-9f55-e6396f883756 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:47.793345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:47.794693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.795084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.795234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.795936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:47.796246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:47.797785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:47.822168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 371/738] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:47.822168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b80575-5056-4984-9c8d-2256317bd6df None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:47.822168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b80575-5056-4984-9c8d-2256317bd6df None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:47.822168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b80575-5056-4984-9c8d-2256317bd6df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:47.822835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 368/739] 10.222.0.77 () {56 vars in 1149 bytes} [Mon Aug 29 22:27:47 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:47.822835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:47.846625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-62b80575-5056-4984-9c8d-2256317bd6df None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:47.847977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-62b80575-5056-4984-9c8d-2256317bd6df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:47.932936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 372/740] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:47.933885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b80575-5056-4984-9c8d-2256317bd6df None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:47.940613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:47.957146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757cb656-3f23-43be-be1c-1233449b7951 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:48.013766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e8e0cf-9c84-425c-951c-5f7f6505d747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:48.015239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8e0cf-9c84-425c-951c-5f7f6505d747 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.016460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8e0cf-9c84-425c-951c-5f7f6505d747 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.017243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8e0cf-9c84-425c-951c-5f7f6505d747 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.020405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e8e0cf-9c84-425c-951c-5f7f6505d747 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:48.023804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:48.027462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e8e0cf-9c84-425c-951c-5f7f6505d747 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:48.052694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 369/741] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:27:47 2022] POST /identity/v3/services => generated 251 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:48.053876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e8e0cf-9c84-425c-951c-5f7f6505d747 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:48.068823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b80575-5056-4984-9c8d-2256317bd6df None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:48.124317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfa45b3-e512-4dec-abb8-d95a960f8636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:48.125356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa45b3-e512-4dec-abb8-d95a960f8636 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.125888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa45b3-e512-4dec-abb8-d95a960f8636 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.126367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa45b3-e512-4dec-abb8-d95a960f8636 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.189473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e8e0cf-9c84-425c-951c-5f7f6505d747 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:48.241412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002975b5-dc40-43bb-bf58-10a589a38141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:48.242090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002975b5-dc40-43bb-bf58-10a589a38141 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.242483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002975b5-dc40-43bb-bf58-10a589a38141 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.242786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002975b5-dc40-43bb-bf58-10a589a38141 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.244464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfa45b3-e512-4dec-abb8-d95a960f8636 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:48.246152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfa45b3-e512-4dec-abb8-d95a960f8636 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:48.256349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002975b5-dc40-43bb-bf58-10a589a38141 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:48.257821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002975b5-dc40-43bb-bf58-10a589a38141 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-002975b5-dc40-43bb-bf58-10a589a38141 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:48.282609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:48.284012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:48.284915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:48.284915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:48.284915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:48.287178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 370/742] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:48.288726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf86cb-117b-4975-8ac2-01bf1106170b None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.289261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf86cb-117b-4975-8ac2-01bf1106170b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.289598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf86cb-117b-4975-8ac2-01bf1106170b None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.291365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 371/743] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:48.292425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf86cb-117b-4975-8ac2-01bf1106170b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:48.295722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:48.340829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:48.398088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 373/744] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:47 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 29 22:27:48.399655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ef053-33a0-4dcc-ac81-66dd71a102df None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.400097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ef053-33a0-4dcc-ac81-66dd71a102df None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.400370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ef053-33a0-4dcc-ac81-66dd71a102df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.421127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-365ef053-33a0-4dcc-ac81-66dd71a102df None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:48.422133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf86cb-117b-4975-8ac2-01bf1106170b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:48.422356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-365ef053-33a0-4dcc-ac81-66dd71a102df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:48.466819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:48.467547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.467887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.468226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.468946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:48.469327 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:48.470890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:48.486616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 372/745] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:48.491683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:48.522764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 374/746] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:48 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 29 22:27:48.522764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365ef053-33a0-4dcc-ac81-66dd71a102df None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:48.569729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:48.612310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37f08e6-12cf-49a4-9a02-ff2b2dbcd72e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:48.652182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70b07b6-4b2c-470f-b9e1-f2136f68f924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:48.652968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b07b6-4b2c-470f-b9e1-f2136f68f924 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.653367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b07b6-4b2c-470f-b9e1-f2136f68f924 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.653728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b07b6-4b2c-470f-b9e1-f2136f68f924 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.654483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70b07b6-4b2c-470f-b9e1-f2136f68f924 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:48.656048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70b07b6-4b2c-470f-b9e1-f2136f68f924 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:48.657776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:48.667343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365ef053-33a0-4dcc-ac81-66dd71a102df None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:48.678882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 373/747] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:27:48 2022] POST /identity/v3/services => generated 240 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:48.680078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70b07b6-4b2c-470f-b9e1-f2136f68f924 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:48.767706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:48.768498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.768619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.768827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.804309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:48.805429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:48.830762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:48.832012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:48.833151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:48.833151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:48.833151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:48.838392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 375/748] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:48.838392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:48.848701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70b07b6-4b2c-470f-b9e1-f2136f68f924 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:48.884372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:48.902988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:48.904570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:48.904570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:48.904570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:48.912778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:48.914247 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:48.932336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:48.933309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:48.935532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 374/749] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:48.938492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.006306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a37f5da-2aa4-4a5b-a455-118742a742ef None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:49.057115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d6dd5-429f-46e8-a260-3f8a4def7303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:49.059169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d6dd5-429f-46e8-a260-3f8a4def7303 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:49.060306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d6dd5-429f-46e8-a260-3f8a4def7303 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:49.060963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d6dd5-429f-46e8-a260-3f8a4def7303 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:49.077357 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746d6dd5-429f-46e8-a260-3f8a4def7303 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-746d6dd5-429f-46e8-a260-3f8a4def7303 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:49.084743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:49.086004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:49.086998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:49.097180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 376/750] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:49.097180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d6dd5-429f-46e8-a260-3f8a4def7303 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.104623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:49.120155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066f1b69-27f8-4d92-b3ce-260f1b9fab14 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:49.165850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:49.166877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:49.167415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:49.167902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:49.169734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:49.170311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:49.172371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:49.188425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 375/751] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:49.189634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.263084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:49.266509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d6dd5-429f-46e8-a260-3f8a4def7303 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:49.323362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:49.323901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:49.324213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:49.324491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:49.325248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:49.325558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:49.327350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:49.343317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bfb1f1-de3b-47fe-bb93-051ebd1667ef None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:49.345174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 377/752] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:49.346149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.393401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:49.394024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:49.394312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:49.394590 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:49.395522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:49.396037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:49.397618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:49.409848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 376/753] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:49.410824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.440994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:49.496605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63709e11-5027-4f1c-837e-47135af27552 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:49.544104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cd9b8e-d778-4f7b-be02-5069f6768a35 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:49.546883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:49.551209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:49.552311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:49.552790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:49.553259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:49.563935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:49.565985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:49.581180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:49.582106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:49.583149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:49.583149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:49.583149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:49.583149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:49.583149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:49.583149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:49.583149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:49.583149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:49.588782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 378/754] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:49.588782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.596359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:49.597054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:49.597508 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:49.597778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:49.598522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=03361f4b51d34cf28c4b5451c3b1f881) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:49.598857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:49.600325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:49.607855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 377/755] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:49 2022] GET /identity/v3/role_assignments?user.id=03361f4b51d34cf28c4b5451c3b1f881&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=38171c89968943bd92d9e9758db65615 => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:49.608521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.727915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6606ab16-a995-47b1-a021-570e5a5187bb None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:49.758442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:49.775449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:49.776546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:49.777040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:49.777516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:49.798843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:49.800787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:49.821978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:49.823006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:49.824445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:49.824445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:49.824445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:49.827499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa3aadb-14a7-4a28-bd2f-2a0d0931ea7b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:49.828091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 379/756] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:49.833404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.866672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:49.873092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:49.873092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:49.873092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:49.873092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:49.873092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:49.873092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:49.892092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 378/757] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:49.895143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:49.962369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:50.031560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d8ef8c-b253-4fc6-aadd-103cdc56f103 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:50.078583 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:50.079254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:50.079519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:50.080435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:50.083312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:50.083702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:50.085042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:50.107171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 380/758] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:50.116031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:50.139341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9689a9d3-2b3e-4026-a0da-937ad57dfd38 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:50.177882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:50.333388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c984a4aa-208d-4820-acee-28b4eee05729 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:50.396229 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:50.399568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:50.400571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:50.402535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:50.403955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=18211b2dc99f43769685855f07c64a3e) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:50.406046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:50.408059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:50.431017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 379/759] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:49 2022] GET /identity/v3/role_assignments?user.id=18211b2dc99f43769685855f07c64a3e&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 259 bytes in 529 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:50.448487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ca4416-c397-467f-9636-6ac740b313c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:50.449166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ca4416-c397-467f-9636-6ac740b313c1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:50.449453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ca4416-c397-467f-9636-6ac740b313c1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:50.449730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ca4416-c397-467f-9636-6ac740b313c1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:50.472896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ca4416-c397-467f-9636-6ac740b313c1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:50.475813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ca4416-c397-467f-9636-6ac740b313c1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-44ca4416-c397-467f-9636-6ac740b313c1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:50.509640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:50.510679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:50.511666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:50.519091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 381/760] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:27:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 403 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:50.520314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:50.583728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe9de95-2e8b-4d6c-80bb-1ee7bee2e8f9 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:50.584232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe9de95-2e8b-4d6c-80bb-1ee7bee2e8f9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:50.584723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe9de95-2e8b-4d6c-80bb-1ee7bee2e8f9 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:50.595380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 382/761] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:50 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:50.604248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe560f93-8b8e-4dd6-a1e1-2545b81cec0a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:50.604749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe560f93-8b8e-4dd6-a1e1-2545b81cec0a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:50.605226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe560f93-8b8e-4dd6-a1e1-2545b81cec0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:50.653183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-fe560f93-8b8e-4dd6-a1e1-2545b81cec0a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:50.657588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-fe560f93-8b8e-4dd6-a1e1-2545b81cec0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:50.691171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:50.704077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8c897a-52c6-4055-97a8-96213efebd0e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:50.746372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:50.747184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:50.748246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:50.748738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:50.749968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:50.750286 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:50.751943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:50.772561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 380/762] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:27:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:50.787986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:50.830543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 383/763] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:27:50.975028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5472159-e53d-42e3-85b8-6ce889439259 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:51.045706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fbc73d-2c91-4c0b-a438-f5d8c99e6e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:51.046833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbc73d-2c91-4c0b-a438-f5d8c99e6e6c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:51.047329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbc73d-2c91-4c0b-a438-f5d8c99e6e6c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:51.047805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbc73d-2c91-4c0b-a438-f5d8c99e6e6c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:51.065659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fbc73d-2c91-4c0b-a438-f5d8c99e6e6c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:51.067164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fbc73d-2c91-4c0b-a438-f5d8c99e6e6c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-76fbc73d-2c91-4c0b-a438-f5d8c99e6e6c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:51.091673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:51.092703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:51.093608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:51.093608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:51.093608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:51.093608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 381/764] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:51.099110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe560f93-8b8e-4dd6-a1e1-2545b81cec0a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:51.165402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:51.252198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ad0c8c-7a95-4024-ab26-1b1f25ce9189 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:51.252589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ad0c8c-7a95-4024-ab26-1b1f25ce9189 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:51.252861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ad0c8c-7a95-4024-ab26-1b1f25ce9189 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:51.254817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 382/765] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:51.263727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c91b03e-6fc7-469b-b9c2-1cac2d3adddb None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:51.264209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c91b03e-6fc7-469b-b9c2-1cac2d3adddb None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:51.264582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c91b03e-6fc7-469b-b9c2-1cac2d3adddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:51.272439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:51.291308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3c91b03e-6fc7-469b-b9c2-1cac2d3adddb None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:51.292557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3c91b03e-6fc7-469b-b9c2-1cac2d3adddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:51.313849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe560f93-8b8e-4dd6-a1e1-2545b81cec0a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:51.390987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:51.391618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:51.391902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:51.392176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:51.392840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:51.393148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:51.394550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:51.424778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 384/766] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:51.431366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:51.445758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 383/767] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:27:51.446868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c91b03e-6fc7-469b-b9c2-1cac2d3adddb None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:51.643447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:51.663749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c91b03e-6fc7-469b-b9c2-1cac2d3adddb None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:51.671972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f9eb8-f2e0-4e8d-b5f3-fe55184e1528 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:51.712875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:51.713624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:51.714016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:51.714297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:51.727457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:51.730082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:51.743835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:51.746073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:51.748273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 385/768] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:51.752491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:51.809002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:51.839834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:51.840790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:51.841756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:51.842141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:51.843086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:51.845249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:51.893298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:51.895325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:51.897144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 384/769] 10.222.0.77 () {56 vars in 1129 bytes} [Mon Aug 29 22:27:51 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 449 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:51.908730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:51.916905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad88ab-3dbe-4e50-baba-c467fa090ce1 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:51.971975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:51.973022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:51.973924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:51.974388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:51.975646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:51.976176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:51.978866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:51.998573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 386/770] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:52.004780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:52.039320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:52.185155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95605fec-a3ba-41dd-8f37-5625c4811f6a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:52.227298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe25d6fb-8de0-4886-a3d6-4c43b64c04ab None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:52.234689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83776b02-98a3-426f-80ff-daea5e53731f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:52.236757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83776b02-98a3-426f-80ff-daea5e53731f None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:52.237309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83776b02-98a3-426f-80ff-daea5e53731f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:52.237800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83776b02-98a3-426f-80ff-daea5e53731f None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/57491e4de6c3435e9d818266a910b831/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:52.241351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83776b02-98a3-426f-80ff-daea5e53731f None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=57491e4de6c3435e9d818266a910b831, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:52.243409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83776b02-98a3-426f-80ff-daea5e53731f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:52.263111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 387/771] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:27:52 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/57491e4de6c3435e9d818266a910b831/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:27:52.264464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83776b02-98a3-426f-80ff-daea5e53731f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:52.277296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee6dde4-6d5c-47df-ab07-39389e127f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:52.277941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee6dde4-6d5c-47df-ab07-39389e127f2d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:52.278226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee6dde4-6d5c-47df-ab07-39389e127f2d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:52.279058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee6dde4-6d5c-47df-ab07-39389e127f2d None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:52.279816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee6dde4-6d5c-47df-ab07-39389e127f2d None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:52.280138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee6dde4-6d5c-47df-ab07-39389e127f2d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:52.281622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee6dde4-6d5c-47df-ab07-39389e127f2d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:52.288739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 385/772] 10.222.0.77 () {56 vars in 1124 bytes} [Mon Aug 29 22:27:51 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 381 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:52.290176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b26a23a-4614-461e-9e4e-36f4d276f87d None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:52.290460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b26a23a-4614-461e-9e4e-36f4d276f87d None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:52.290783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b26a23a-4614-461e-9e4e-36f4d276f87d None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:52.299221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 386/773] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:52.299221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b26a23a-4614-461e-9e4e-36f4d276f87d None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:52.381656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:52.403016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:52.435712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83776b02-98a3-426f-80ff-daea5e53731f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:52.477470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b26a23a-4614-461e-9e4e-36f4d276f87d None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:52.489656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e26016-ed38-48cc-875a-37682139a4ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:52.490310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e26016-ed38-48cc-875a-37682139a4ea None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:52.490616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e26016-ed38-48cc-875a-37682139a4ea None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:52.490961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e26016-ed38-48cc-875a-37682139a4ea None admin] PATH_INFO: `/v3/services/network` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:52.491536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e26016-ed38-48cc-875a-37682139a4ea None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:52.492938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e26016-ed38-48cc-875a-37682139a4ea None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-98e26016-ed38-48cc-875a-37682139a4ea None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:52.503493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:52.505743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:52.507766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 388/774] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:52 2022] GET /identity/v3/services/network => generated 88 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:52.509585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c60406f-9775-4107-9be2-2eed77ea6a45 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:52.509585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c60406f-9775-4107-9be2-2eed77ea6a45 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:52.509585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c60406f-9775-4107-9be2-2eed77ea6a45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:52.545546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-0c60406f-9775-4107-9be2-2eed77ea6a45 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:52.547701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:52.548450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:52.549036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:52.549425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0c60406f-9775-4107-9be2-2eed77ea6a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:52.549634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:52.550600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:52.551016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:52.552604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:52.565604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 387/775] 10.222.0.77 () {56 vars in 1124 bytes} [Mon Aug 29 22:27:52 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:52.566649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:52.686271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 389/776] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:27:52.686271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c60406f-9775-4107-9be2-2eed77ea6a45 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:52.748871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e28a75-2e6d-4270-8ec3-b9e3a8e792ca None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:52.787158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:52.812858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:52.843558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:52.844479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:52.844926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:52.845351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:52.846152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] RBAC: Adding query filter params (name=network) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:52.846665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:52.848488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:52.855483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c60406f-9775-4107-9be2-2eed77ea6a45 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:52.861649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 388/777] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:27:52 2022] GET /identity/v3/services?name=network => generated 124 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:52.867721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:52.941170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:52.941709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:52.941986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:52.942259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] PATH_INFO: `/v3/endpoints` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:52.943043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] RBAC: Adding query filter params (interface=public, service_id=fe0c0d0f2a174a63a989ebf929325df3, region_id=RegionOne) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:52.943359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:52.944770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:52.958460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 390/778] 10.222.0.77 () {56 vars in 1251 bytes} [Mon Aug 29 22:27:52 2022] GET /identity/v3/endpoints?service_id=fe0c0d0f2a174a63a989ebf929325df3&interface=public®ion_id=RegionOne => generated 194 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:52.965013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:53.063669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f307f-11b8-44c2-b9a2-43becbe50d64 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:53.119090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:53.120029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.120314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.120592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.121516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] RBAC: Adding query filter params (type=network) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:53.121827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:53.124296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:53.134932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 389/779] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:27:52 2022] GET /identity/v3/services?type=network => generated 350 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:53.135874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:53.145107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4464f-cd00-43b6-97f1-62f5a8436fcf None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:53.172791 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:53.186782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7099fe-142c-44f0-819f-cb3e0c4c2556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:53.187557 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7099fe-142c-44f0-819f-cb3e0c4c2556 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.187940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7099fe-142c-44f0-819f-cb3e0c4c2556 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.188314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7099fe-142c-44f0-819f-cb3e0c4c2556 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.189046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7099fe-142c-44f0-819f-cb3e0c4c2556 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:53.190701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7099fe-142c-44f0-819f-cb3e0c4c2556 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:53.197328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 391/780] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:27:52 2022] GET /identity/v3/services => generated 772 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:53.198261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7099fe-142c-44f0-819f-cb3e0c4c2556 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:53.217476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:53.292214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78a2ce6-feb7-426d-90a3-035f831d7b19 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:53.335594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869092c4-47cd-4035-8b96-19ce9801da84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:53.336622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869092c4-47cd-4035-8b96-19ce9801da84 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.337119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869092c4-47cd-4035-8b96-19ce9801da84 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.337587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869092c4-47cd-4035-8b96-19ce9801da84 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.351626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869092c4-47cd-4035-8b96-19ce9801da84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-869092c4-47cd-4035-8b96-19ce9801da84 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:53.358390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:53.359683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:53.360550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:53.361241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7099fe-142c-44f0-819f-cb3e0c4c2556 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:53.367186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 390/781] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:53.368931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ece11-d0c5-44cc-be2c-6b12c9a8480b None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.369425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ece11-d0c5-44cc-be2c-6b12c9a8480b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.369903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ece11-d0c5-44cc-be2c-6b12c9a8480b None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.373938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 391/782] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:53 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:53.375087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891ece11-d0c5-44cc-be2c-6b12c9a8480b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:53.408110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec59c6f-1a9a-466f-b924-cd995bf56231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:53.409540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec59c6f-1a9a-466f-b924-cd995bf56231 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.410028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec59c6f-1a9a-466f-b924-cd995bf56231 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.410538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec59c6f-1a9a-466f-b924-cd995bf56231 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.411492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec59c6f-1a9a-466f-b924-cd995bf56231 None admin] RBAC: Adding query filter params (interface=public, service_id=459a980b739244e8a8642dcc40ba6225, region_id=RegionOne) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:53.412142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec59c6f-1a9a-466f-b924-cd995bf56231 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:53.413914 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec59c6f-1a9a-466f-b924-cd995bf56231 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:53.420714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 392/783] 10.222.0.77 () {56 vars in 1251 bytes} [Mon Aug 29 22:27:53 2022] GET /identity/v3/endpoints?service_id=459a980b739244e8a8642dcc40ba6225&interface=public®ion_id=RegionOne => generated 194 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:53.422245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a16dc-d587-4900-91b4-52c7bd158691 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.422836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a16dc-d587-4900-91b4-52c7bd158691 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.423146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a16dc-d587-4900-91b4-52c7bd158691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.443275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a93a16dc-d587-4900-91b4-52c7bd158691 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:53.444811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a93a16dc-d587-4900-91b4-52c7bd158691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:53.518843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891ece11-d0c5-44cc-be2c-6b12c9a8480b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:53.524069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:53.538774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:53.554063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 393/784] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:27:53.555882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93a16dc-d587-4900-91b4-52c7bd158691 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:53.570571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:53.571646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.572503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.572963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.573972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:53.574565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:53.577230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:53.591145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 392/785] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:53.602758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:53.787721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93a16dc-d587-4900-91b4-52c7bd158691 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:53.870182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e030ce-5367-41d3-9518-4c08d6cb491b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:53.884825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d525bfe-fa1c-4704-a0b3-7f1023c23459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:53.885461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d525bfe-fa1c-4704-a0b3-7f1023c23459 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.885744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d525bfe-fa1c-4704-a0b3-7f1023c23459 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.886019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d525bfe-fa1c-4704-a0b3-7f1023c23459 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.886973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d525bfe-fa1c-4704-a0b3-7f1023c23459 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:53.889667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d525bfe-fa1c-4704-a0b3-7f1023c23459 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:53.898509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 394/786] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:27:53 2022] GET /identity/v3/services => generated 772 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:53.900011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d525bfe-fa1c-4704-a0b3-7f1023c23459 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:53.912346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:53.913394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:53.913878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:53.914340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:53.926483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:53.928364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:53.961040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:53.962083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:53.967679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:53.967679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:53.967679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:53.969324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 393/787] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 371 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:53.976585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:53.979202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:54.040949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:54.113112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d525bfe-fa1c-4704-a0b3-7f1023c23459 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:54.154301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4dd39-090c-4df8-9911-00ee298a41d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:54.154972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4dd39-090c-4df8-9911-00ee298a41d7 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:54.155264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4dd39-090c-4df8-9911-00ee298a41d7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:54.155551 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4dd39-090c-4df8-9911-00ee298a41d7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:54.167539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af4dd39-090c-4df8-9911-00ee298a41d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-2af4dd39-090c-4df8-9911-00ee298a41d7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:54.173112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:54.174442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:54.177066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 395/788] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:27:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:54.182775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4dd39-090c-4df8-9911-00ee298a41d7 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:54.233733 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cb44b4-3dd2-49b6-9ae6-ca8de61e079a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:54.311669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4dd39-090c-4df8-9911-00ee298a41d7 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:54.322485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:54.351913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:54.352958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:54.353501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:54.353959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:54.355437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:54.355725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:54.358582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:54.369398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:54.370080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:54.370366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:54.370639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:54.371371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] RBAC: Adding query filter params (name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:54.371721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:54.373188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:54.383227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 396/789] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:27:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:54.388090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 394/790] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 412 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:54.393011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:54.411210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:54.576676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:54.598460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b1f637-251a-431f-8b11-c1ce385a469d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:54.634426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c08b737-d27c-41f9-8ebb-3b1fe43f7b06 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:54.684236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c809092-4fbb-43b1-ba0b-5f1cbe295b09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:54.685862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c809092-4fbb-43b1-ba0b-5f1cbe295b09 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:54.686349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c809092-4fbb-43b1-ba0b-5f1cbe295b09 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:54.686865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c809092-4fbb-43b1-ba0b-5f1cbe295b09 None admin] PATH_INFO: `/v3/users/swift` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:54.690118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:54.690756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:54.691112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:54.691390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:54.705609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:54.707083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c809092-4fbb-43b1-ba0b-5f1cbe295b09 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:54.707290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:54.709042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c809092-4fbb-43b1-ba0b-5f1cbe295b09 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-5c809092-4fbb-43b1-ba0b-5f1cbe295b09 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:54.735623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:54.736637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:27:54.737596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:54.738156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:54.739074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:54.741185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:54.741185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:54.741185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:54.743124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 395/791] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:54.743547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 397/792] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:54.744522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:54.745262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f644d47e-c751-4000-a478-770cb476af23 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:54.745750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f644d47e-c751-4000-a478-770cb476af23 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:54.746209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f644d47e-c751-4000-a478-770cb476af23 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:54.748715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 396/793] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:54.754794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2dc75-0ee1-47c0-b2f3-7c51ffdbb3c2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:54.754794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2dc75-0ee1-47c0-b2f3-7c51ffdbb3c2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:54.754794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2dc75-0ee1-47c0-b2f3-7c51ffdbb3c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:54.773219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:54.779441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c0c2dc75-0ee1-47c0-b2f3-7c51ffdbb3c2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:54.781093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c0c2dc75-0ee1-47c0-b2f3-7c51ffdbb3c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:54.872020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1155c7-f672-4bdd-8ea9-7cfcf14d33f4 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:54.882843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 397/794] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:27:54.884011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c2dc75-0ee1-47c0-b2f3-7c51ffdbb3c2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:54.954154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:54.955143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:54.956082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:54.956510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:54.958395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:54.959069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:54.962107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:54.993604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 398/795] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:55.001532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:55.013114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:55.065476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c2dc75-0ee1-47c0-b2f3-7c51ffdbb3c2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:55.132201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:55.133267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:55.136005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:55.136444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:55.137583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:55.139178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:55.140652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:55.174077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 398/796] 10.222.0.77 () {56 vars in 1135 bytes} [Mon Aug 29 22:27:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:55.175296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:55.179716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:55.222088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a780739c-00e1-466a-8f52-e3b04dcd2300 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:55.271069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:55.272953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:55.273422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:55.273893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:55.291405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:55.293164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:55.309316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:55.310544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:27:55.311534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:55.312516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 399/797] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:27:55 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:55.313388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:55.321275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7b3344-b277-4b39-91bf-887182266400 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:55.373131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:55.374370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:55.375257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:55.375752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:55.378375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:55.392774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:55.395500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:55.413530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:55.414575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:55.421207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:55.421207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:55.421207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:55.421207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 399/798] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:55.421207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:55.451249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288fd9a4-3d08-4d18-a336-0fc982874427 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:55.492117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5025540c-6f32-4c49-ab4e-b1364f6f6d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:55.492930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025540c-6f32-4c49-ab4e-b1364f6f6d5c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:55.493324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025540c-6f32-4c49-ab4e-b1364f6f6d5c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:55.493661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025540c-6f32-4c49-ab4e-b1364f6f6d5c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:55.503728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:55.507814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5025540c-6f32-4c49-ab4e-b1364f6f6d5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5025540c-6f32-4c49-ab4e-b1364f6f6d5c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:55.512778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:55.514699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:55.515784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:55.516836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 400/799] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:55.518024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5025540c-6f32-4c49-ab4e-b1364f6f6d5c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:55.549825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb5018e-d6ec-4ff6-a985-a9534e3b6c9f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:55.594099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:55.594998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:55.595457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:55.595846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:55.596798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:55.597491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:55.599301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:55.623271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 400/800] 10.222.0.77 () {56 vars in 1149 bytes} [Mon Aug 29 22:27:55 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:55.624828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:55.657215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5025540c-6f32-4c49-ab4e-b1364f6f6d5c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:55.710237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:55.720352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:55.721164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:55.721545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:55.722455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:55.723270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:55.723708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:55.725535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:55.744232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 401/801] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:55.745925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:55.818084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c9f76a-19f0-45fc-8774-d05103ef35ae None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:55.852196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:55.867107 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:55.868098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:55.868619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:55.869123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:55.870054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:55.870574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:55.872175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:55.887765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 401/802] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:55.889643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:55.911352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e501140-d80b-443a-bca2-b20626b7eca7 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:56.022937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d8dfe9-6a98-4397-9577-43986548e635 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:56.033483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:56.076507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37952a6e-a6fb-4e4b-997a-c00a908f6bbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:56.077342 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37952a6e-a6fb-4e4b-997a-c00a908f6bbf None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.077695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:56.077836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37952a6e-a6fb-4e4b-997a-c00a908f6bbf None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.078232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37952a6e-a6fb-4e4b-997a-c00a908f6bbf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.078388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.078592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.078897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.087583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:56.088702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:56.091219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37952a6e-a6fb-4e4b-997a-c00a908f6bbf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:56.094796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37952a6e-a6fb-4e4b-997a-c00a908f6bbf None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:56.105017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:56.105988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:56.107536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 402/803] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:56.108548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-37952a6e-a6fb-4e4b-997a-c00a908f6bbf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:56.122169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:56.132948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 402/804] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 378 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f18155-ecf9-4088-86a3-c405b1e6cd98 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f18155-ecf9-4088-86a3-c405b1e6cd98 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f18155-ecf9-4088-86a3-c405b1e6cd98 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 403/805] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:56.134820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f18155-ecf9-4088-86a3-c405b1e6cd98 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:56.263724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9887f27-9f6b-4890-820b-940ac057af1e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:56.264643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f18155-ecf9-4088-86a3-c405b1e6cd98 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:56.282382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:56.312493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4883bb8-6cbc-4966-b9b2-d2e1739346de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:56.313144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:56.313285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4883bb8-6cbc-4966-b9b2-d2e1739346de None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.313386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4883bb8-6cbc-4966-b9b2-d2e1739346de None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.313639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4883bb8-6cbc-4966-b9b2-d2e1739346de None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.313743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.314128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.314348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4883bb8-6cbc-4966-b9b2-d2e1739346de None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:56.314523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.314665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4883bb8-6cbc-4966-b9b2-d2e1739346de None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:56.316143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4883bb8-6cbc-4966-b9b2-d2e1739346de None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:56.332155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:56.334252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:56.338413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 404/806] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:56.339627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791edc6b-4f96-4e4c-b9f9-e53e79a5c354 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.339915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791edc6b-4f96-4e4c-b9f9-e53e79a5c354 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.340187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791edc6b-4f96-4e4c-b9f9-e53e79a5c354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:56.354105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:56.357255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:56.358543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:56.358543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:56.358543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:56.358543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 403/807] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:56.358543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:56.364848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-791edc6b-4f96-4e4c-b9f9-e53e79a5c354 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:56.366040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-791edc6b-4f96-4e4c-b9f9-e53e79a5c354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:56.390461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:56.454758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 405/808] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:27:56.455603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791edc6b-4f96-4e4c-b9f9-e53e79a5c354 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:56.506833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b590d-5088-4680-9e8f-a868c75d5f1a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:56.550686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:56.551690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.552139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.552410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.553062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:56.553415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:56.554925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:56.568645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 404/809] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:56.569558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:56.582446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791edc6b-4f96-4e4c-b9f9-e53e79a5c354 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:56.584362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:56.623421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe38c8a-3ca1-4d7a-abab-4674f7517293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:56.624101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe38c8a-3ca1-4d7a-abab-4674f7517293 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.624381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe38c8a-3ca1-4d7a-abab-4674f7517293 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.624661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe38c8a-3ca1-4d7a-abab-4674f7517293 None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/03361f4b51d34cf28c4b5451c3b1f881/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.626699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe38c8a-3ca1-4d7a-abab-4674f7517293 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=03361f4b51d34cf28c4b5451c3b1f881, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:56.628171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe38c8a-3ca1-4d7a-abab-4674f7517293 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:56.645450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 406/810] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:27:56 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/03361f4b51d34cf28c4b5451c3b1f881/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:56.653752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a701b35-f9fa-4724-81f3-750a647cd586 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.654403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a701b35-f9fa-4724-81f3-750a647cd586 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.655043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a701b35-f9fa-4724-81f3-750a647cd586 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.657142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 407/811] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:56 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:56.658376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a701b35-f9fa-4724-81f3-750a647cd586 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:56.668119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:56.697292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3711a42-1a6c-4657-8838-d4edb09b64a8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:56.743073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc1b722-f77d-43aa-9dd8-735e8ac837ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:56.743695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1b722-f77d-43aa-9dd8-735e8ac837ee None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.743964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1b722-f77d-43aa-9dd8-735e8ac837ee None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.744270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc1b722-f77d-43aa-9dd8-735e8ac837ee None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.744942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc1b722-f77d-43aa-9dd8-735e8ac837ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:56.745242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc1b722-f77d-43aa-9dd8-735e8ac837ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:56.746644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc1b722-f77d-43aa-9dd8-735e8ac837ee None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:56.767475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 405/812] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:56.770518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb35f3f-dc61-42e6-9923-7989906d6bcf None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.771159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb35f3f-dc61-42e6-9923-7989906d6bcf None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.771740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb35f3f-dc61-42e6-9923-7989906d6bcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.801392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-acb35f3f-dc61-42e6-9923-7989906d6bcf None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:56.802677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-acb35f3f-dc61-42e6-9923-7989906d6bcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:56.826782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a701b35-f9fa-4724-81f3-750a647cd586 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:56.866373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:56.867101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:56.867384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:56.867737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:56.875751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:56.877209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:56.882547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 406/813] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:27:56.883261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb35f3f-dc61-42e6-9923-7989906d6bcf None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:56.899435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:56.900575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:56.901654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:56.901654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:56.901654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:56.901654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:56.901654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:56.901654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:56.901654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:56.901654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:56.905283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 408/814] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:56.906666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:56.944195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:56.978000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:57.020011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb35f3f-dc61-42e6-9923-7989906d6bcf None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.062173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.062922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.063303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.063679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.064384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:57.065968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:57.076090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:57.078363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 29 22:27:57.080250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:57.083452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 407/815] 10.222.0.77 () {56 vars in 1129 bytes} [Mon Aug 29 22:27:56 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:57.084500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:57.109008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90625465-e145-49f3-b907-d6a5f071d8ad None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.155358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.156317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.157089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.157089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] PATH_INFO: `/v3/users/nova` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.167555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:57.169016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:27:57.182299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:57.183413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:27:57.184361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:57.186943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 409/816] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:27:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:57.188146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:57.261315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05957e54-3417-4853-bcdf-eb37e84dc4c4 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.305273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:57.307531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:57.314122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.314672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.315011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.315245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.316459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:57.316934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:57.318397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:57.331068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db65387-fb43-44e8-9ccf-5bf4fdb1992a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.331854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 408/817] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:57.332052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:57.385904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.386826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.387731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.388198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.389461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:57.389936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:57.391855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:57.398052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 410/818] 10.222.0.77 () {56 vars in 1124 bytes} [Mon Aug 29 22:27:57 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:57.399092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:57.483277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1527119e-b99a-407d-8bc5-0f29bd9ee961 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.524220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.524907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.525274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.525552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] PATH_INFO: `/v3/services/network` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.526252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:57.527853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:57.538337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:57.540568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 29 22:27:57.542421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:57.545128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 409/819] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:57 2022] GET /identity/v3/services/network => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:57.546414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:57.551627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9666066c-deb3-426a-a027-78ab04d00e1f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.609752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:57.616167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.617301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.617689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.618169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.619541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:57.619919 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:57.622361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:57.658775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 411/820] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:27:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:57.658775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:57.661410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:57.671242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac2bb58-5d1f-4088-8006-80fde5a1f53f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.733889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9755649f-a28f-4fcb-89ce-62dcb1b0c7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.735524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9755649f-a28f-4fcb-89ce-62dcb1b0c7a4 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.736257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9755649f-a28f-4fcb-89ce-62dcb1b0c7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.736986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9755649f-a28f-4fcb-89ce-62dcb1b0c7a4 None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/18211b2dc99f43769685855f07c64a3e/roles/6c40acc5170649d8ad758d42c85b91e0` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.755615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9755649f-a28f-4fcb-89ce-62dcb1b0c7a4 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=18211b2dc99f43769685855f07c64a3e, role_id=6c40acc5170649d8ad758d42c85b91e0)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:57.757085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9755649f-a28f-4fcb-89ce-62dcb1b0c7a4 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:57.776486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 410/821] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:27:57 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/18211b2dc99f43769685855f07c64a3e/roles/6c40acc5170649d8ad758d42c85b91e0 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:27:57.777606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9755649f-a28f-4fcb-89ce-62dcb1b0c7a4 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:57.846579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb492419-9f97-4589-9bbb-926147529537 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.903116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.904623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.905084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.905536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.907380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:57.907972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:57.909851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:57.919395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 412/822] 10.222.0.77 () {56 vars in 1124 bytes} [Mon Aug 29 22:27:57 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:57.921146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:57.943412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9755649f-a28f-4fcb-89ce-62dcb1b0c7a4 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:57.953418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:57.995555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:57.996549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:57.996830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:57.997473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:57.997894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] RBAC: Adding query filter params (name=network) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:57.998618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:58.002243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:58.009800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 411/823] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:27:57 2022] GET /identity/v3/services?name=network => generated 124 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:58.013710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:58.045705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:58.098890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaeb8a8-9b4f-4275-9688-54636bd3c3e8 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:58.142209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d71fed-9194-4b42-bfdb-7e188ad0647e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:58.142924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d71fed-9194-4b42-bfdb-7e188ad0647e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:58.143221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d71fed-9194-4b42-bfdb-7e188ad0647e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:58.143478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d71fed-9194-4b42-bfdb-7e188ad0647e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:58.194800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fc5147-e15a-49e1-a8ad-47f0abb2dcd5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:58.235533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d71fed-9194-4b42-bfdb-7e188ad0647e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:58.237119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d71fed-9194-4b42-bfdb-7e188ad0647e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:58.278321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d431a7-16ff-4d17-aa9f-9f7566b0962a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:58.280148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d431a7-16ff-4d17-aa9f-9f7566b0962a None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:58.282280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d431a7-16ff-4d17-aa9f-9f7566b0962a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:58.282978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d431a7-16ff-4d17-aa9f-9f7566b0962a None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:58.284675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d431a7-16ff-4d17-aa9f-9f7566b0962a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:58.288480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d431a7-16ff-4d17-aa9f-9f7566b0962a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:58.323588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:58.337017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 412/824] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:27:58 2022] POST /identity/v3/endpoints => generated 346 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:27:58.342880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d431a7-16ff-4d17-aa9f-9f7566b0962a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:58.393841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:58.408856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 413/825] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:27:57 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:27:58.416885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d71fed-9194-4b42-bfdb-7e188ad0647e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:58.548561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d71fed-9194-4b42-bfdb-7e188ad0647e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:58.590143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d431a7-16ff-4d17-aa9f-9f7566b0962a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:58.594637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:58.595354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:58.595636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:58.595906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:58.611860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:58.614215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:27:58.634816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:58.639403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:27:58.641851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:27:58.641851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:27:58.641851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:58.641851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 414/826] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:27:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:58.641851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:58.645782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:58.647075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:58.647794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:58.648439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:58.649786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] RBAC: Adding query filter params (type=network) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:58.650487 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:58.653343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:58.666353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 413/827] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:27:58 2022] GET /identity/v3/services?type=network => generated 350 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:58.667637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:58.669862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:58.781594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:58.803783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b6a9de-dfa8-4acb-839e-ef242c5bd2f5 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:58.806715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e643aa-7405-4c20-81e4-a722afbe34de None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:58.847162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585e1b93-515e-43aa-98bb-a41507f5a9e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:58.848478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e1b93-515e-43aa-98bb-a41507f5a9e8 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:58.849060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e1b93-515e-43aa-98bb-a41507f5a9e8 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:58.849502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e1b93-515e-43aa-98bb-a41507f5a9e8 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:58.850313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585e1b93-515e-43aa-98bb-a41507f5a9e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:58.850789 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585e1b93-515e-43aa-98bb-a41507f5a9e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:58.852647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585e1b93-515e-43aa-98bb-a41507f5a9e8 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:58.868375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 415/828] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:27:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:27:58.870003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f02a9-65ef-4164-87be-74486310f496 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:58.870464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f02a9-65ef-4164-87be-74486310f496 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:58.870925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f02a9-65ef-4164-87be-74486310f496 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:58.873030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec43043-4920-4b67-b3d1-41ab3edd27b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:58.873415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 416/829] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:27:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:27:58.873957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec43043-4920-4b67-b3d1-41ab3edd27b0 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:58.874448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec43043-4920-4b67-b3d1-41ab3edd27b0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:58.880547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec43043-4920-4b67-b3d1-41ab3edd27b0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:58.880714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545f02a9-65ef-4164-87be-74486310f496 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:58.880796 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec43043-4920-4b67-b3d1-41ab3edd27b0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:58.881821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec43043-4920-4b67-b3d1-41ab3edd27b0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:58.907270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 414/830] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:27:58 2022] POST /identity/v3/endpoints => generated 337 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:27:58.910208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ee722-3b85-492d-b979-bde80a776dba None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:58.910720 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ee722-3b85-492d-b979-bde80a776dba None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:58.911222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ee722-3b85-492d-b979-bde80a776dba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:58.959099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-374ee722-3b85-492d-b979-bde80a776dba None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:27:58.960789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-374ee722-3b85-492d-b979-bde80a776dba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:27:59.035604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:59.096331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545f02a9-65ef-4164-87be-74486310f496 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:59.142017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 415/831] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:27:58 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:27:59.164892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:59.165552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:59.165838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:59.166153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:59.174050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:59.175587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:27:59.188294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:59.190258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:27:59.192148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 417/832] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:27:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:27:59.196013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374ee722-3b85-492d-b979-bde80a776dba None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:59.289346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:59.329589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374ee722-3b85-492d-b979-bde80a776dba None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:59.384783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:59.385342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:59.385672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:59.385955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:59.386681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:59.387023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:59.390306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:59.407147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 416/833] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:27:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:59.410229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:59.531114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:59.569972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76510a88-ae63-4d55-bfa3-427b23316a32 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:59.629533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:59.630598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:59.631078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:59.631361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:59.632247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=57491e4de6c3435e9d818266a910b831) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:27:59.632565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:27:59.634502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:27:59.639019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:59.646817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 418/834] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:27:59 2022] GET /identity/v3/role_assignments?user.id=57491e4de6c3435e9d818266a910b831&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=38171c89968943bd92d9e9758db65615 => generated 608 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:27:59.698820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1be15e2-582d-4a7d-864a-3d9b25f9e953 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:27:59.746365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:27:59.756044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120f1a57-688d-41f0-bb9d-f62094bffe10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:27:59.757115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f1a57-688d-41f0-bb9d-f62094bffe10 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:27:59.757583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f1a57-688d-41f0-bb9d-f62094bffe10 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:27:59.758012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f1a57-688d-41f0-bb9d-f62094bffe10 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:27:59.772737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120f1a57-688d-41f0-bb9d-f62094bffe10 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-120f1a57-688d-41f0-bb9d-f62094bffe10 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:27:59.780350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:27:59.781586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:27:59.783160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:27:59.787308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 417/835] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:27:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:27:59.798090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:27:59.987992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a607678-c2b9-477b-a9e6-aa1a78a575d4 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:00.033340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56fdfd7-71eb-49c9-8558-ca9dd253a9b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:00.033960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56fdfd7-71eb-49c9-8558-ca9dd253a9b5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:00.034502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56fdfd7-71eb-49c9-8558-ca9dd253a9b5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:00.034845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56fdfd7-71eb-49c9-8558-ca9dd253a9b5 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:00.035555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56fdfd7-71eb-49c9-8558-ca9dd253a9b5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:00.035855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56fdfd7-71eb-49c9-8558-ca9dd253a9b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:00.037506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56fdfd7-71eb-49c9-8558-ca9dd253a9b5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:00.048445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 419/836] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:27:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:00.055903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120f1a57-688d-41f0-bb9d-f62094bffe10 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:00.175126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120f1a57-688d-41f0-bb9d-f62094bffe10 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:00.212295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:00.212954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:00.213269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:00.213580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:00.226071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:00.227893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:00.240935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b3874-16dd-4087-a350-be647234abbf None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:00.241330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b3874-16dd-4087-a350-be647234abbf None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:00.241804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b3874-16dd-4087-a350-be647234abbf None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:00.243743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:00.244683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:00.245598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:00.245598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:00.245598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:00.245598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 418/838] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:00.249934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 420/838] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:00.252197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:00.254251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1aebc1-27e5-4af4-bf12-c3cf3cd84b14 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:00.254718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1aebc1-27e5-4af4-bf12-c3cf3cd84b14 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:00.255230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1aebc1-27e5-4af4-bf12-c3cf3cd84b14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:00.288199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-9e1aebc1-27e5-4af4-bf12-c3cf3cd84b14 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:00.290457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9e1aebc1-27e5-4af4-bf12-c3cf3cd84b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:00.321483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:00.383942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:00.414266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 421/839] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:00 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:00.415185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08da0767-9eb3-4ed2-9036-5eb0ad1debb7 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:00.453532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:00.454161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:00.454444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:00.454715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:00.455399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:00.455705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:00.457104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:00.470772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 419/840] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:00.478965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1aebc1-27e5-4af4-bf12-c3cf3cd84b14 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:00.612147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1aebc1-27e5-4af4-bf12-c3cf3cd84b14 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:00.656230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:00.657286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:00.657763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:00.658240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] PATH_INFO: `/v3/users/swift` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:00.666645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:00.668359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:00.681243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:00.682323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 29 22:28:00.683331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:00.686464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 422/841] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:28:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:00.693139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:00.819458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68790ab-f06a-4fb2-a9ec-b068171113bc None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:00.823821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:00.883140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:00.884011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:00.884500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:00.884968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:00.886280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:00.886865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:00.889614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:00.923113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 420/842] 10.222.0.77 () {56 vars in 1135 bytes} [Mon Aug 29 22:28:00 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:00.932511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:00.939800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:01.023265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:01.098453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86723fb-ab31-4347-a477-d13bb016f7e4 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:01.102459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd4b46-8394-40a7-8220-b65e96e79cda None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:01.157943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5661b970-5ce2-4b18-9c9d-c052d8391a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:01.158846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5661b970-5ce2-4b18-9c9d-c052d8391a00 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:01.159207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5661b970-5ce2-4b18-9c9d-c052d8391a00 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:01.159952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5661b970-5ce2-4b18-9c9d-c052d8391a00 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:01.161298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4252303-9073-4aa8-bd9b-5c648303ade0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:01.162298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4252303-9073-4aa8-bd9b-5c648303ade0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:01.162762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4252303-9073-4aa8-bd9b-5c648303ade0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:01.163227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4252303-9073-4aa8-bd9b-5c648303ade0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:01.183014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4252303-9073-4aa8-bd9b-5c648303ade0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-a4252303-9073-4aa8-bd9b-5c648303ade0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:01.191509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:01.193507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 421/843] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:28:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:01.196214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23540c8d-4917-4e0e-a880-bdab37a7d1d9 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:01.198852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23540c8d-4917-4e0e-a880-bdab37a7d1d9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:01.198852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23540c8d-4917-4e0e-a880-bdab37a7d1d9 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:01.199695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 422/844] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:01.200840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23540c8d-4917-4e0e-a880-bdab37a7d1d9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:01.237634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:01.313812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5661b970-5ce2-4b18-9c9d-c052d8391a00 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:01.316511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5661b970-5ce2-4b18-9c9d-c052d8391a00 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:01.365130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23540c8d-4917-4e0e-a880-bdab37a7d1d9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:01.403466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:01.417396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a03da5e-1faa-4dfc-8a6f-22b0f914c5ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:01.418320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a03da5e-1faa-4dfc-8a6f-22b0f914c5ae None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:01.418841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a03da5e-1faa-4dfc-8a6f-22b0f914c5ae None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:01.419339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a03da5e-1faa-4dfc-8a6f-22b0f914c5ae None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:01.422311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a03da5e-1faa-4dfc-8a6f-22b0f914c5ae None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:01.422311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a03da5e-1faa-4dfc-8a6f-22b0f914c5ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:01.424644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a03da5e-1faa-4dfc-8a6f-22b0f914c5ae None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:01.446791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 423/845] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:28:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:01.446791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fd325b-f5c0-4997-9e6e-1cbc056cf49a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:01.446791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fd325b-f5c0-4997-9e6e-1cbc056cf49a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:01.446791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fd325b-f5c0-4997-9e6e-1cbc056cf49a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:01.465298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-50fd325b-f5c0-4997-9e6e-1cbc056cf49a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:01.466582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-50fd325b-f5c0-4997-9e6e-1cbc056cf49a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:01.487366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 423/846] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:00 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 556 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:28:01.492680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5661b970-5ce2-4b18-9c9d-c052d8391a00 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:01.565259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 424/847] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:01 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:01.566821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fd325b-f5c0-4997-9e6e-1cbc056cf49a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:01.603897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:01.655146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5661b970-5ce2-4b18-9c9d-c052d8391a00 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:01.702169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:01.702748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:01.703030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:01.703300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:01.715977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:01.718839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:01.733495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fd325b-f5c0-4997-9e6e-1cbc056cf49a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:01.735538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:01.736765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:01.737683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:01.737683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:01.737683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:01.737683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 424/848] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:01.746594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:01.754273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:01.808579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:01.809596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:01.810082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:01.811127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:01.835349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:01.838917 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:01.861646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:01.863554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:01.865375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:01.865375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:01.865375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:01.869051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 425/849] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:01.878804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:01.895991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20c4f58-d903-421a-a56e-4bb180cd300d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:01.958587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:01.959286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:01.959765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:01.960031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:01.960931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:01.961237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:01.962663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:01.986768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 425/850] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:01.986768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:01.987765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:02.045457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6cd9ae-285d-4cfc-ade4-9b9032cef429 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.081057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:02.108054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:02.108640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.109094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.109595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.110809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:02.111342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:02.113979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:02.131216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 426/851] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:02.132077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:02.138207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80f85d-246e-4c50-9ebd-ab1c4928076c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.179569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc77add6-cbe8-4e24-90d6-4940b27ed4ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:02.180468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77add6-cbe8-4e24-90d6-4940b27ed4ef None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.180942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77add6-cbe8-4e24-90d6-4940b27ed4ef None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.181623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77add6-cbe8-4e24-90d6-4940b27ed4ef None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.182243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc77add6-cbe8-4e24-90d6-4940b27ed4ef None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:02.183787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc77add6-cbe8-4e24-90d6-4940b27ed4ef None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-cc77add6-cbe8-4e24-90d6-4940b27ed4ef None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:02.196227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:02.197289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:02.198219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 426/852] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:01 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:02.199118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b7e26-a722-4f61-a989-a23dc516701c None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.199118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b7e26-a722-4f61-a989-a23dc516701c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.199118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b7e26-a722-4f61-a989-a23dc516701c None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.200487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 427/853] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:02 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:02.201565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6b7e26-a722-4f61-a989-a23dc516701c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:02.266308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d0246c-6a60-4c82-b4cf-dada36592d2a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.298397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:02.310895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:02.311613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.311931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.312247 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.322044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:02.324295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:02.330697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6b7e26-a722-4f61-a989-a23dc516701c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:02.345236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:02.346577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 427/854] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:28:02 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:02.349172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:02.369181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:02.381247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76020ab5-d3d0-49d5-abf3-2aa3afeea2fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:02.382460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76020ab5-d3d0-49d5-abf3-2aa3afeea2fb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.382962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76020ab5-d3d0-49d5-abf3-2aa3afeea2fb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.383441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76020ab5-d3d0-49d5-abf3-2aa3afeea2fb None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.396914 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76020ab5-d3d0-49d5-abf3-2aa3afeea2fb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:02.399772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76020ab5-d3d0-49d5-abf3-2aa3afeea2fb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-76020ab5-d3d0-49d5-abf3-2aa3afeea2fb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:02.422445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:02.424207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:02.426070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 428/855] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:02.427156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91681ddc-ee20-4f27-8c7f-71887844f4e5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.427156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91681ddc-ee20-4f27-8c7f-71887844f4e5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.427925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91681ddc-ee20-4f27-8c7f-71887844f4e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.449009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-91681ddc-ee20-4f27-8c7f-71887844f4e5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:02.450226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-91681ddc-ee20-4f27-8c7f-71887844f4e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:02.510182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd9b219-1e92-4ea9-85a6-c13e3ae85fe3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.551496 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:02.552105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.552388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.553154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.553425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:02.553982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:02.555239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:02.568235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 428/856] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:02 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:02.568435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 429/857] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:02 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:02.568902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:02.568998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91681ddc-ee20-4f27-8c7f-71887844f4e5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:02.655324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:02.712459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91681ddc-ee20-4f27-8c7f-71887844f4e5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.729181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d770046-38ff-4062-9c17-2005f9f54c01 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.735877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:02.756546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:02.756934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.757214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.757482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.758187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:02.758490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:02.759979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:02.782420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 430/858] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:28:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:02.782420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:02.787320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:02.787949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.788227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.788502 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.789667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:02.791247 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:02.792842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:02.832353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 429/859] 10.222.0.77 () {56 vars in 1149 bytes} [Mon Aug 29 22:28:02 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:02.832353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:02.918279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a55bb6-480c-4020-b65b-bb3c0791b0fb None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.961028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4f4382-24ec-499e-98d2-5eeb664f4fa0 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:02.967745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:02.968676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:02.969117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:02.969554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:02.970616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:02.971160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:02.973550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:02.977475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:02.990797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 431/860] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:02 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:02.990797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:03.030474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:03.031351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:03.031750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:03.032174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:03.033127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=03361f4b51d34cf28c4b5451c3b1f881) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:03.033604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:03.035624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:03.053794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 430/861] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:02 2022] GET /identity/v3/role_assignments?user.id=03361f4b51d34cf28c4b5451c3b1f881&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=38171c89968943bd92d9e9758db65615 => generated 608 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:03.053794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:03.069915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:03.180804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51075fbc-ba57-4d8b-b086-1e9b8be3f924 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:03.212936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d72ef5-e789-4003-9655-e76ecf2c5a6e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:03.233698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a481d7e-09cd-4c69-9fcd-b9c0f191bf8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:03.234845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a481d7e-09cd-4c69-9fcd-b9c0f191bf8f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:03.235361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a481d7e-09cd-4c69-9fcd-b9c0f191bf8f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:03.235848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a481d7e-09cd-4c69-9fcd-b9c0f191bf8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:03.254582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:03.255572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:03.256018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:03.256712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:03.257458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:03.259930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:03.276176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:03.277210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:03.278122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:03.278948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 431/862] 10.222.0.77 () {56 vars in 1132 bytes} [Mon Aug 29 22:28:03 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:03.286759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:03.343536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a481d7e-09cd-4c69-9fcd-b9c0f191bf8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:03.346543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:03.348351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a481d7e-09cd-4c69-9fcd-b9c0f191bf8f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:03.412555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:03.450490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f991e-e937-411b-a7f5-7d1bd6a741d6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:03.510964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 432/863] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:02 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 529 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:28:03.518449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:03.518449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:03.518449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:03.518449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:03.518971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:03.518971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:03.518971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:03.520593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a481d7e-09cd-4c69-9fcd-b9c0f191bf8f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:03.529900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 432/864] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:03 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:03.535902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:03.697084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:03.703537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567f4897-7b32-4ce6-a127-edc0d5f7a97d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:03.711449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a481d7e-09cd-4c69-9fcd-b9c0f191bf8f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:03.763107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:03.765226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:03.765546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:03.766189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:03.766717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:03.767054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:03.767373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:03.767931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:03.769167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:03.769786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:03.772466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:03.784061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:03.789660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 433/865] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:03 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:03.792775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:03.796871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:03.832842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:03.834682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:03.836538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:03.836538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:03.836538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:03.839741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 433/866] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:03.846875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:03.989863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e6a088-b705-47c4-af8b-a2e391ab302e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:04.050293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:04.050935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:04.051218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:04.051494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:04.052161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:04.052467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:04.053871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:04.070965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 434/867] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:04.074155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:04.225641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7778d4-4590-48eb-8202-7f6c93cb152e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:04.264520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:04.265505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:04.265999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:04.266465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:04.274175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:04.276041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:04.289656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:04.290623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:04.291931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:04.291931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:04.291931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:04.291931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:04.291931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:04.291931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:04.291931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:04.291931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:04.294332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 434/868] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:04.302866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:04.341308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:04.433376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50577410-fcef-4b7e-bff0-44761610ca75 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:04.472807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a31ad40-a964-4673-80b3-17911b966628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:04.473429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a31ad40-a964-4673-80b3-17911b966628 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:04.473708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a31ad40-a964-4673-80b3-17911b966628 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:04.473981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a31ad40-a964-4673-80b3-17911b966628 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:04.474690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a31ad40-a964-4673-80b3-17911b966628 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:04.475032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a31ad40-a964-4673-80b3-17911b966628 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:04.476830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a31ad40-a964-4673-80b3-17911b966628 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:04.496940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 435/869] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:28:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:04.499502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:04.580091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:04.639756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff2145-f3ef-44a0-a0e4-e5c580f9ecc7 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:04.679812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a9fbca-66b5-4f7c-9a1b-797a069e45d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:04.680470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9fbca-66b5-4f7c-9a1b-797a069e45d4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:04.680750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9fbca-66b5-4f7c-9a1b-797a069e45d4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:04.681018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9fbca-66b5-4f7c-9a1b-797a069e45d4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:04.681749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a9fbca-66b5-4f7c-9a1b-797a069e45d4 None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=18211b2dc99f43769685855f07c64a3e) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:04.682058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a9fbca-66b5-4f7c-9a1b-797a069e45d4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:04.683520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a9fbca-66b5-4f7c-9a1b-797a069e45d4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:04.689978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 435/870] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:04 2022] GET /identity/v3/role_assignments?user.id=18211b2dc99f43769685855f07c64a3e&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 608 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:05.452389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8c1109-9f5b-44c9-ba21-70f307912717 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:05.453401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8c1109-9f5b-44c9-ba21-70f307912717 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:05.453899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8c1109-9f5b-44c9-ba21-70f307912717 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:05.456092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 436/871] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:05.462092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b0dd2-bf32-408a-a27e-caa9c3b23414 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:05.463063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b0dd2-bf32-408a-a27e-caa9c3b23414 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:05.463544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969b0dd2-bf32-408a-a27e-caa9c3b23414 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:05.483966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-969b0dd2-bf32-408a-a27e-caa9c3b23414 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:05.485787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-969b0dd2-bf32-408a-a27e-caa9c3b23414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:05.594834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 436/872] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:05 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:05.614356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4161d15d-d2c3-4be7-834e-9c95054b5f0e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:05.614786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4161d15d-d2c3-4be7-834e-9c95054b5f0e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:05.615039 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4161d15d-d2c3-4be7-834e-9c95054b5f0e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:05.616752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 437/873] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:05.624289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e58577e-908c-41a4-9b8f-dbe5c88d3c94 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:05.624928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e58577e-908c-41a4-9b8f-dbe5c88d3c94 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:05.625405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e58577e-908c-41a4-9b8f-dbe5c88d3c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:05.653976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7e58577e-908c-41a4-9b8f-dbe5c88d3c94 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:05.656266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7e58577e-908c-41a4-9b8f-dbe5c88d3c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:05.723452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:05.748766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 437/874] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:05 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:06.033304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4161d15d-d2c3-4be7-834e-9c95054b5f0e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:06.128379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e58577e-908c-41a4-9b8f-dbe5c88d3c94 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:06.177899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:06.228906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4161d15d-d2c3-4be7-834e-9c95054b5f0e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:06.303192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:06.305527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:06.306045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:06.306545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:06.308582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:06.314585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:06.332536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:06.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:06.336304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 438/875] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:06 2022] GET /identity/v3/services/object-store => generated 93 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:06.343379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:06.435684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e58577e-908c-41a4-9b8f-dbe5c88d3c94 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:06.527026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6854ed0b-ee22-41e3-8b40-766fed3bab16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:06.528032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6854ed0b-ee22-41e3-8b40-766fed3bab16 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:06.528502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6854ed0b-ee22-41e3-8b40-766fed3bab16 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:06.528951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6854ed0b-ee22-41e3-8b40-766fed3bab16 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:06.531027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6854ed0b-ee22-41e3-8b40-766fed3bab16 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:06.534035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6854ed0b-ee22-41e3-8b40-766fed3bab16 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:06.552004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45239872-8440-4831-8c00-6dedfd3b0996 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:06.558097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 438/876] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:28:06 2022] POST /identity/v3/services => generated 251 bytes in 429 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:06.558936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f52fd-6cd7-4320-83c9-ef3c333789c9 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:06.560183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f52fd-6cd7-4320-83c9-ef3c333789c9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:06.560183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f52fd-6cd7-4320-83c9-ef3c333789c9 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:06.566822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 439/877] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:06.571090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ae8347-31d1-4fca-a527-f12baf95d0f5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:06.571683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ae8347-31d1-4fca-a527-f12baf95d0f5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:06.571683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ae8347-31d1-4fca-a527-f12baf95d0f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:06.595251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e2ae8347-31d1-4fca-a527-f12baf95d0f5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:06.596891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e2ae8347-31d1-4fca-a527-f12baf95d0f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:06.617997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:06.618784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:06.619160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:06.619451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:06.620313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:06.620712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:06.622398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:06.628955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 439/878] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:06 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:06.635149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:06.645909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:06.648720 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:06.711447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 440/879] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:06 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:06.773842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a711b9-5341-4943-ab6f-15f7312ba361 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:06.814765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bfada8-7ac4-4c84-9e81-d912c962615f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:06.815301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfada8-7ac4-4c84-9e81-d912c962615f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:06.815572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfada8-7ac4-4c84-9e81-d912c962615f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:06.815847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfada8-7ac4-4c84-9e81-d912c962615f None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:06.816493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bfada8-7ac4-4c84-9e81-d912c962615f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:06.816801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bfada8-7ac4-4c84-9e81-d912c962615f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:06.818211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bfada8-7ac4-4c84-9e81-d912c962615f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:06.824593 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 440/880] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:06 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:07.235725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ae8347-31d1-4fca-a527-f12baf95d0f5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:07.380648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b0722-2fe8-4d95-9329-a41e7bf3b3e1 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:07.381472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b0722-2fe8-4d95-9329-a41e7bf3b3e1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:07.382195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b0722-2fe8-4d95-9329-a41e7bf3b3e1 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:07.385539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 441/881] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:07.391546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965774e3-0505-49fd-a4f6-b7c54c2159ae None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:07.392071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965774e3-0505-49fd-a4f6-b7c54c2159ae None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:07.392675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965774e3-0505-49fd-a4f6-b7c54c2159ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:07.405375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ae8347-31d1-4fca-a527-f12baf95d0f5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:07.420964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-965774e3-0505-49fd-a4f6-b7c54c2159ae None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:07.424174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-965774e3-0505-49fd-a4f6-b7c54c2159ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:07.465473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e489e7e-e553-4471-93e9-d83f9402d5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:07.466348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e489e7e-e553-4471-93e9-d83f9402d5cd None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:07.466636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e489e7e-e553-4471-93e9-d83f9402d5cd None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:07.467568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e489e7e-e553-4471-93e9-d83f9402d5cd None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:07.468830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e489e7e-e553-4471-93e9-d83f9402d5cd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:07.471159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e489e7e-e553-4471-93e9-d83f9402d5cd None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:07.502864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 441/882] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:28:07 2022] POST /identity/v3/services => generated 269 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:07.548389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:07.557496 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 442/883] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:07 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:07.884110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e489e7e-e553-4471-93e9-d83f9402d5cd None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:07.939740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:08.058725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e489e7e-e553-4471-93e9-d83f9402d5cd None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:08.109454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37514bac-5800-4e6a-82d2-cfa7f700d377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:08.110965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37514bac-5800-4e6a-82d2-cfa7f700d377 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:08.111517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37514bac-5800-4e6a-82d2-cfa7f700d377 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:08.112117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37514bac-5800-4e6a-82d2-cfa7f700d377 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:08.128848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37514bac-5800-4e6a-82d2-cfa7f700d377 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-37514bac-5800-4e6a-82d2-cfa7f700d377 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:08.135108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:08.136244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:08.137183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:08.140721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 442/884] 10.222.0.77 () {56 vars in 1120 bytes} [Mon Aug 29 22:28:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:08.150027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965774e3-0505-49fd-a4f6-b7c54c2159ae None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:08.294517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965774e3-0505-49fd-a4f6-b7c54c2159ae None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:08.361882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:08.362885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:08.363385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:08.363868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:08.365091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:08.365655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:08.368325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:08.381428 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 443/885] 10.222.0.77 () {56 vars in 1115 bytes} [Mon Aug 29 22:28:08 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:08.387141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37514bac-5800-4e6a-82d2-cfa7f700d377 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:08.489205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:08.503651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37514bac-5800-4e6a-82d2-cfa7f700d377 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:08.541175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:08.542189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:08.542691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:08.543193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:08.559385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:08.561913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:08.581701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:08.582716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:08.583666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:08.583666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:08.583666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:08.583666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 443/886] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:08.589133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:08.677338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:08.755191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfef099-b5aa-469d-8a3c-d37bc42f504a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:08.830310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749d7e14-0f20-4667-8975-151638741196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:08.831386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d7e14-0f20-4667-8975-151638741196 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:08.831873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d7e14-0f20-4667-8975-151638741196 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:08.832349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d7e14-0f20-4667-8975-151638741196 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:08.833601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749d7e14-0f20-4667-8975-151638741196 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:08.834122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749d7e14-0f20-4667-8975-151638741196 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:08.836896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749d7e14-0f20-4667-8975-151638741196 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:08.866937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 444/887] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:08.872882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:08.956018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2ae2c-345e-4626-a42f-fe6ee5263438 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:08.956267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2ae2c-345e-4626-a42f-fe6ee5263438 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:08.956587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2ae2c-345e-4626-a42f-fe6ee5263438 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:08.958256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 445/888] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:08.964549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d6ccd-f4b1-4945-a431-693ca5dff3a1 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:08.964895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d6ccd-f4b1-4945-a431-693ca5dff3a1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:08.965279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d6ccd-f4b1-4945-a431-693ca5dff3a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:08.986710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-599d6ccd-f4b1-4945-a431-693ca5dff3a1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:08.988211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-599d6ccd-f4b1-4945-a431-693ca5dff3a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:09.009243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4578ec5a-6f62-4921-a6e4-9f2a4208f47e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:09.054667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:09.056061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:09.056790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:09.057440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:09.069573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:09.071619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:09.082918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 446/889] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:08 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:09.083250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:09.090663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:09.091907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:28:09.093502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:09.095890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 444/890] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:28:08 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:09.104360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599d6ccd-f4b1-4945-a431-693ca5dff3a1 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:09.171311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:09.244535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599d6ccd-f4b1-4945-a431-693ca5dff3a1 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:09.317919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:09.319096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:09.319617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:09.320350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:09.321592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:09.322117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:09.325057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:09.363377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 447/891] 10.222.0.77 () {56 vars in 1149 bytes} [Mon Aug 29 22:28:09 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:09.374176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:09.468921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:09.563776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8df493-f47a-4efe-8d55-32e74898578d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:09.607087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3741f2d7-1b0b-4e6c-a8c2-da1403a66c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:09.608810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3741f2d7-1b0b-4e6c-a8c2-da1403a66c2f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:09.609350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3741f2d7-1b0b-4e6c-a8c2-da1403a66c2f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:09.609861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3741f2d7-1b0b-4e6c-a8c2-da1403a66c2f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:09.714409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:09.715167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de240a47-fc07-4063-b844-0d1d6312fd73 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:09.720190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3741f2d7-1b0b-4e6c-a8c2-da1403a66c2f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:09.721388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3741f2d7-1b0b-4e6c-a8c2-da1403a66c2f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:09.730263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:09.772795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17918387-5779-4e7b-99dd-6213f453904e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:09.773861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17918387-5779-4e7b-99dd-6213f453904e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:09.774375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17918387-5779-4e7b-99dd-6213f453904e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:09.774886 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17918387-5779-4e7b-99dd-6213f453904e None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:09.775729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17918387-5779-4e7b-99dd-6213f453904e None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:09.777240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17918387-5779-4e7b-99dd-6213f453904e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-17918387-5779-4e7b-99dd-6213f453904e None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:09.792155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:09.794340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:09.797875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 448/892] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:09 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:09.799723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88b4c3c-3aa3-4442-a628-b35fea8ecf89 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:09.799723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88b4c3c-3aa3-4442-a628-b35fea8ecf89 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:09.799723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88b4c3c-3aa3-4442-a628-b35fea8ecf89 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:09.799723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 449/893] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:09.803622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0729c4a-b652-4e64-b444-aced3cf67148 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:09.804481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0729c4a-b652-4e64-b444-aced3cf67148 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:09.804481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0729c4a-b652-4e64-b444-aced3cf67148 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:09.832485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b0729c4a-b652-4e64-b444-aced3cf67148 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:09.833737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b0729c4a-b652-4e64-b444-aced3cf67148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:09.869226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 445/894] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:09 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:28:09.870577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3741f2d7-1b0b-4e6c-a8c2-da1403a66c2f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:09.941751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 450/895] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:09 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:09.942823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0729c4a-b652-4e64-b444-aced3cf67148 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:10.049126 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:10.055968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3741f2d7-1b0b-4e6c-a8c2-da1403a66c2f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:10.121127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:10.122531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.123684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.124211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.125363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:10.126188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:10.128739 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:10.130422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0729c4a-b652-4e64-b444-aced3cf67148 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:10.132324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:10.138585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 446/896] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:09 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:10.144478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:10.185402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:10.186192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.186479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.186775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.200426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:10.202074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:10.223289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:10.224398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:10.225293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:10.225293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:10.225293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:10.229572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 451/897] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:10.230586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:10.286723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e89b2-45c3-4db7-8967-8a33a53fc053 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:10.336693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54af651-a422-4010-903f-c94dc4489b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:10.337827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54af651-a422-4010-903f-c94dc4489b22 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.338313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54af651-a422-4010-903f-c94dc4489b22 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.338844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54af651-a422-4010-903f-c94dc4489b22 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.340010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54af651-a422-4010-903f-c94dc4489b22 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:10.340594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54af651-a422-4010-903f-c94dc4489b22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:10.344163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54af651-a422-4010-903f-c94dc4489b22 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:10.361000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 447/898] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:10 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:10.361000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f62463-7bbd-4ebb-aa57-59fd2c30425b None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.361000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f62463-7bbd-4ebb-aa57-59fd2c30425b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.361000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f62463-7bbd-4ebb-aa57-59fd2c30425b None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.362199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 448/899] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:10.363243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f62463-7bbd-4ebb-aa57-59fd2c30425b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:10.377894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e79ce9e-25b4-4199-8df8-0240e659145f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:10.401796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:10.423914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:10.424531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.424811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.425089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.425775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:10.426081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:10.427518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:10.441983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 452/900] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:10.442898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:10.465880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:10.519069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f62463-7bbd-4ebb-aa57-59fd2c30425b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:10.564111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691653cd-10b8-4d26-a382-791ad5d4ed04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:10.565353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691653cd-10b8-4d26-a382-791ad5d4ed04 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.565640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691653cd-10b8-4d26-a382-791ad5d4ed04 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.565918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691653cd-10b8-4d26-a382-791ad5d4ed04 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.566613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691653cd-10b8-4d26-a382-791ad5d4ed04 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:10.568053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691653cd-10b8-4d26-a382-791ad5d4ed04 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:10.579446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf561db-c9d9-4266-bc8c-51b9a97cec45 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:10.587462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 449/901] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:28:10 2022] POST /identity/v3/services => generated 241 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:10.589871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521cb290-eb2d-4c93-9568-ad83dfb5a8d5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.590356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521cb290-eb2d-4c93-9568-ad83dfb5a8d5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.590869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521cb290-eb2d-4c93-9568-ad83dfb5a8d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.620480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-521cb290-eb2d-4c93-9568-ad83dfb5a8d5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:10.622390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-521cb290-eb2d-4c93-9568-ad83dfb5a8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:10.643011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:10.643895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.644385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.644883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.646023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] RBAC: Adding query filter params (interface=public, service_id=62e7a85a96cf4e48b9233c71d67a4cbf, region_id=RegionOne) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:10.646581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:10.649432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:10.664995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 453/902] 10.222.0.77 () {56 vars in 1251 bytes} [Mon Aug 29 22:28:10 2022] GET /identity/v3/endpoints?service_id=62e7a85a96cf4e48b9233c71d67a4cbf&interface=public®ion_id=RegionOne => generated 194 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:10.664995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:10.747103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 450/903] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:10.748558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521cb290-eb2d-4c93-9568-ad83dfb5a8d5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:10.787084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:10.831527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:10.885827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99b09c2-2bfd-4c1a-8e95-09ee607a99ec None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:10.934611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:10.935418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:10.936421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:10.936802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:10.942309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521cb290-eb2d-4c93-9568-ad83dfb5a8d5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:10.945790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:10.947707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:10.967963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:10.969031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:10.970530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:10.970530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:10.970530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:10.970530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:10.970530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:10.970530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:10.970530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:10.970530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:10.974660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 454/904] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:10.976620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:11.009578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41c2ca2-d81c-480c-bdb1-9beaa9555ede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:11.010721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41c2ca2-d81c-480c-bdb1-9beaa9555ede None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:11.011410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41c2ca2-d81c-480c-bdb1-9beaa9555ede None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:11.011912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41c2ca2-d81c-480c-bdb1-9beaa9555ede None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:11.012957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41c2ca2-d81c-480c-bdb1-9beaa9555ede None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:11.015338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41c2ca2-d81c-480c-bdb1-9beaa9555ede None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:11.024134 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 451/905] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:28:10 2022] GET /identity/v3/services => generated 1497 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:28:11.093799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41c2ca2-d81c-480c-bdb1-9beaa9555ede None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:11.186937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edda20ee-cb5f-4dc6-8fbe-505c413b7b1c None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:11.263876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:11.265350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:11.265813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:11.266472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:11.267597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:11.268768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:11.269412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:11.271972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:11.275889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:11.295797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 455/906] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:28:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:11.305773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:11.310759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41c2ca2-d81c-480c-bdb1-9beaa9555ede None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:11.361631 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:11.362471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:11.362914 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:11.363219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:11.363915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:11.365321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:11.377000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:11.378029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:11.378983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 452/907] 10.222.0.77 () {56 vars in 1132 bytes} [Mon Aug 29 22:28:11 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:11.384965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:11.504119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17869634-c290-4b9e-997f-1a61f16dd696 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:11.509519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f2d82d-b3f5-459c-ad4e-741a76f0ca6b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:11.554567 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:11.555282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:11.555563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:11.555939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:11.556612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:11.556914 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:11.558367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:11.570305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 453/908] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:11 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:11.571442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:11.577806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:11.578907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:11.579396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:11.579873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:11.581269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] RBAC: Adding query filter params (role.id=f772879e7c8a4395a3cdc078d84cc832, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=18211b2dc99f43769685855f07c64a3e) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:11.581968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:11.584983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:11.587011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:11.599273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 456/909] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:11 2022] GET /identity/v3/role_assignments?user.id=18211b2dc99f43769685855f07c64a3e&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=f772879e7c8a4395a3cdc078d84cc832 => generated 259 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:11.743097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fa159b-249e-4b9a-bdde-b5046aa399c9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:11.791930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96466389-dc13-4444-ab95-6127e377146f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:11.792957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96466389-dc13-4444-ab95-6127e377146f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:11.793497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96466389-dc13-4444-ab95-6127e377146f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:11.794004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96466389-dc13-4444-ab95-6127e377146f None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:11.794902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96466389-dc13-4444-ab95-6127e377146f None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:11.796086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96466389-dc13-4444-ab95-6127e377146f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:11.797684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96466389-dc13-4444-ab95-6127e377146f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:11.803997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 454/910] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:11 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:11.813508 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:11.921793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:12.004896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c60f5-8443-4769-ab1a-bef3dfcbd2de None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:12.052157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03673c62-4977-4499-bb8b-1dcd6a6589f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:12.052693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03673c62-4977-4499-bb8b-1dcd6a6589f3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:12.052976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03673c62-4977-4499-bb8b-1dcd6a6589f3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:12.053253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03673c62-4977-4499-bb8b-1dcd6a6589f3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:12.053969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03673c62-4977-4499-bb8b-1dcd6a6589f3 None admin] RBAC: Adding query filter params (interface=public, service_id=1f86ec4072f846cdbcd402cebc361288, region_id=RegionOne) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:12.054283 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03673c62-4977-4499-bb8b-1dcd6a6589f3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:12.055710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03673c62-4977-4499-bb8b-1dcd6a6589f3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:12.062902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 457/911] 10.222.0.77 () {56 vars in 1251 bytes} [Mon Aug 29 22:28:11 2022] GET /identity/v3/endpoints?service_id=1f86ec4072f846cdbcd402cebc361288&interface=public®ion_id=RegionOne => generated 194 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:12.072343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96466389-dc13-4444-ab95-6127e377146f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:12.193101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:12.223062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96466389-dc13-4444-ab95-6127e377146f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:12.263432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07f90e4-1600-4481-b681-6cd2e1c903bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:12.264692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f90e4-1600-4481-b681-6cd2e1c903bf None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:12.264989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f90e4-1600-4481-b681-6cd2e1c903bf None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:12.265271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f90e4-1600-4481-b681-6cd2e1c903bf None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:12.265899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07f90e4-1600-4481-b681-6cd2e1c903bf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:12.267322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07f90e4-1600-4481-b681-6cd2e1c903bf None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:12.274598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 455/912] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:28:12 2022] GET /identity/v3/services => generated 1497 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:28:13.005305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf19153-755c-4f92-98ec-50add75b0d95 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.006111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf19153-755c-4f92-98ec-50add75b0d95 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.006111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf19153-755c-4f92-98ec-50add75b0d95 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.007597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 458/913] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:13.014926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc9d8a0-d011-4b6c-9ecc-f76737b165ee None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.015405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc9d8a0-d011-4b6c-9ecc-f76737b165ee None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.015708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc9d8a0-d011-4b6c-9ecc-f76737b165ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.038677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2bc9d8a0-d011-4b6c-9ecc-f76737b165ee None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:13.039945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2bc9d8a0-d011-4b6c-9ecc-f76737b165ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:13.131657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 456/914] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:13 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:13.443962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf19153-755c-4f92-98ec-50add75b0d95 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:13.492133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8752d089-b492-4a3b-b819-2c1543c43d9b None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.492872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8752d089-b492-4a3b-b819-2c1543c43d9b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.493179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8752d089-b492-4a3b-b819-2c1543c43d9b None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.495356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 457/915] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:13.501277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325dfab-52a9-4c29-8f9a-3668c0fd0056 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.501617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325dfab-52a9-4c29-8f9a-3668c0fd0056 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.501835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325dfab-52a9-4c29-8f9a-3668c0fd0056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.521124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1325dfab-52a9-4c29-8f9a-3668c0fd0056 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:13.522360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1325dfab-52a9-4c29-8f9a-3668c0fd0056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:13.563293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf19153-755c-4f92-98ec-50add75b0d95 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:13.590741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:13.604366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:13.605099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.605317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:13.605515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.605647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.606236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:13.607723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:13.619070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 458/916] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:13 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:13.620616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:13.621576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:13.622830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 459/917] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:13 2022] GET /identity/v3/services/object-store => generated 93 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:13.627919 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1325dfab-52a9-4c29-8f9a-3668c0fd0056 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:13.747006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1325dfab-52a9-4c29-8f9a-3668c0fd0056 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:13.784991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8133720-1f55-49ec-8f14-26ff5e4d6e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:13.785886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8133720-1f55-49ec-8f14-26ff5e4d6e15 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.786162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8133720-1f55-49ec-8f14-26ff5e4d6e15 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.786438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8133720-1f55-49ec-8f14-26ff5e4d6e15 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.787126 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8133720-1f55-49ec-8f14-26ff5e4d6e15 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:13.787434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8133720-1f55-49ec-8f14-26ff5e4d6e15 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:13.788832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8133720-1f55-49ec-8f14-26ff5e4d6e15 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:13.794530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 459/918] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:13 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:13.800377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:13.867613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07811d82-58fe-4ebf-9f36-19cc66bba247 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.868271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07811d82-58fe-4ebf-9f36-19cc66bba247 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.868747 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07811d82-58fe-4ebf-9f36-19cc66bba247 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.871862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 460/919] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:13.878331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1c3781-ae17-459c-be8e-1b5ee6077ac6 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.879026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1c3781-ae17-459c-be8e-1b5ee6077ac6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.879498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1c3781-ae17-459c-be8e-1b5ee6077ac6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.900033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-dc1c3781-ae17-459c-be8e-1b5ee6077ac6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:13.901546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dc1c3781-ae17-459c-be8e-1b5ee6077ac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:13.922780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ed4ff4-03d9-46e1-9f3c-0048d80491c5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:13.959183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:13.959969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:13.960445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:13.960847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:13.961826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:13.962259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:13.963968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:13.969068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:13.977301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 460/920] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:13 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:13.977301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:13.988830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 461/921] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:13 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:13.996296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1c3781-ae17-459c-be8e-1b5ee6077ac6 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:14.067341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:14.157413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd5a699-c0f5-458a-b6fb-42a852ee747f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:14.207224 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1c3781-ae17-459c-be8e-1b5ee6077ac6 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:14.218011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:14.219416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.220076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.220751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.221958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:14.224623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:14.242153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:14.244313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 29 22:28:14.246558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:14.252200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 461/922] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:13 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:14.258597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:14.275044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:14.276089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.276578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.277079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.278065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] RBAC: Adding query filter params (interface=public, service_id=4214798a2d2d463b989437959295ddd2, region_id=RegionOne) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:14.278600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:14.280764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:14.287924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 462/923] 10.222.0.77 () {56 vars in 1251 bytes} [Mon Aug 29 22:28:13 2022] GET /identity/v3/endpoints?service_id=4214798a2d2d463b989437959295ddd2&interface=public®ion_id=RegionOne => generated 194 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:14.296977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:14.390502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:14.415240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9019573-c64c-4401-b977-674a22df5c98 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:14.441652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:14.464440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e642d634-bdf7-44d1-8641-90501afa00a9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:14.476757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:14.477806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.478336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.478980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.479777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:14.480250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:14.482598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:14.494178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 462/924] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:14 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:14.494178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:14.520531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01cdb4e-f800-4f6e-9cdf-e1f864389c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:14.521304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01cdb4e-f800-4f6e-9cdf-e1f864389c41 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.521674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01cdb4e-f800-4f6e-9cdf-e1f864389c41 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.522020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01cdb4e-f800-4f6e-9cdf-e1f864389c41 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.522708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01cdb4e-f800-4f6e-9cdf-e1f864389c41 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:14.524732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01cdb4e-f800-4f6e-9cdf-e1f864389c41 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:14.535860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 463/925] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:28:14 2022] GET /identity/v3/services => generated 1497 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:28:14.548628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01cdb4e-f800-4f6e-9cdf-e1f864389c41 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:14.679722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2e5ef6-fb60-4dda-8d49-2aac9e492a44 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:14.714960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01cdb4e-f800-4f6e-9cdf-e1f864389c41 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:14.746346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:14.749744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8853e1-1a47-44f7-a24a-cb0d11341791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:14.752074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8853e1-1a47-44f7-a24a-cb0d11341791 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.752577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8853e1-1a47-44f7-a24a-cb0d11341791 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.753147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8853e1-1a47-44f7-a24a-cb0d11341791 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.769445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6679c02a-de97-4211-a445-1d4740de014f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:14.770066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6679c02a-de97-4211-a445-1d4740de014f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.770346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6679c02a-de97-4211-a445-1d4740de014f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.770614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6679c02a-de97-4211-a445-1d4740de014f None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.771249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8853e1-1a47-44f7-a24a-cb0d11341791 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:14.771361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6679c02a-de97-4211-a445-1d4740de014f None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:14.771590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6679c02a-de97-4211-a445-1d4740de014f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:14.772960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6679c02a-de97-4211-a445-1d4740de014f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-6d8853e1-1a47-44f7-a24a-cb0d11341791 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:14.776979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:14.777959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:14.778867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:14.780685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 464/926] 10.222.0.77 () {56 vars in 1114 bytes} [Mon Aug 29 22:28:14 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:14.780685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930b19b6-ffee-4069-8383-8041708c6e31 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.780685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930b19b6-ffee-4069-8383-8041708c6e31 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.780921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930b19b6-ffee-4069-8383-8041708c6e31 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.782667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 463/927] 10.222.0.77 () {56 vars in 1120 bytes} [Mon Aug 29 22:28:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:14.785819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8853e1-1a47-44f7-a24a-cb0d11341791 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:14.786805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 465/928] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:14 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:14.788574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d560f35d-838b-45e1-a1fb-a430cbc3fb26 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.788867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d560f35d-838b-45e1-a1fb-a430cbc3fb26 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.789144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d560f35d-838b-45e1-a1fb-a430cbc3fb26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.814281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d560f35d-838b-45e1-a1fb-a430cbc3fb26 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:14.817931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d560f35d-838b-45e1-a1fb-a430cbc3fb26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:14.841829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:14.906781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 466/929] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:14.906781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d560f35d-838b-45e1-a1fb-a430cbc3fb26 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:14.914109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8853e1-1a47-44f7-a24a-cb0d11341791 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:14.958600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ecc280-f995-4391-9a3b-350aee81631e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:14.959337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ecc280-f995-4391-9a3b-350aee81631e None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:14.959814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ecc280-f995-4391-9a3b-350aee81631e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:14.960153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ecc280-f995-4391-9a3b-350aee81631e None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:14.960918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ecc280-f995-4391-9a3b-350aee81631e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:14.962343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ecc280-f995-4391-9a3b-350aee81631e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:14.998977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 464/930] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:28:14 2022] POST /identity/v3/endpoints => generated 346 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:15.073768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d560f35d-838b-45e1-a1fb-a430cbc3fb26 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:15.098024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:15.135992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:15.137191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:15.137880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:15.138550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:15.140203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:15.140831 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:15.142894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:15.156744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 467/931] 10.222.0.77 () {56 vars in 1115 bytes} [Mon Aug 29 22:28:14 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:15.163948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ecc280-f995-4391-9a3b-350aee81631e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:15.255452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:15.320075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:15.335558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ecc280-f995-4391-9a3b-350aee81631e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:15.385776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aed5051-601c-4798-9278-9bc8034faa97 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:15.396746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:15.397791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:15.398272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:15.398758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:15.419740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:15.422846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:15.429981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:15.431034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:15.431531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:15.432011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:15.432843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:15.434447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:15.445537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:15.446569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 29 22:28:15.447608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:15.450504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 468/932] 10.222.0.77 () {56 vars in 1132 bytes} [Mon Aug 29 22:28:15 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:15.450694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:15.451942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:15.452944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:15.452944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:15.452944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:15.453422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 465/933] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:15.462188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:15.463235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:15.512540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:15.586656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87b2ddf-ad3a-4792-a44d-d803e9164e83 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:15.600869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba8bd98-9c2b-49c7-904d-4793691ef64a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:15.636316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:15.637594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:15.638297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:15.639012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:15.640298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:15.640966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:15.643280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:15.661397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 466/934] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:15.669751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:15.669751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:15.669751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:15.669751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:15.670161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:15.670301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:15.670301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:15.672047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:15.676830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:15.684512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 469/935] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:15 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:15.692264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:15.813632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7984de54-d5a8-4e1c-87a7-550a7e674a88 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:15.890913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:15.892840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:15.893522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:15.894675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:15.902639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e0cbec-7088-4664-a575-a0436364c2c2 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:15.907251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:15.914384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:15.921236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:15.945093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:15.946869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:28:15.948222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:15.948824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 467/936] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:28:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:15.956844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:15.974936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:15.975950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:15.976422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:15.976896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:15.978024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:15.978564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:15.981422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:15.999048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 470/937] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:15 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:16.002715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:16.047104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:16.088927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9083c243-abff-4677-b8c8-e09dafd1f887 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:16.133640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e6145-f754-419c-8f3e-2b834cbfc2c2 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:16.136715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:16.137345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:16.137627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:16.137972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:16.139104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:16.140068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:16.141739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:16.175429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 468/938] 10.222.0.77 () {56 vars in 1149 bytes} [Mon Aug 29 22:28:15 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:16.176344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473eeb10-b222-41ef-a5b2-3b81e2d28321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:16.176577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473eeb10-b222-41ef-a5b2-3b81e2d28321 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:16.176858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473eeb10-b222-41ef-a5b2-3b81e2d28321 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:16.177161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473eeb10-b222-41ef-a5b2-3b81e2d28321 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:16.177811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473eeb10-b222-41ef-a5b2-3b81e2d28321 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:16.185261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473eeb10-b222-41ef-a5b2-3b81e2d28321 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:16.188362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:16.207755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 471/939] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:28:16 2022] POST /identity/v3/endpoints => generated 347 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:28:16.357949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc3c38-9d0d-43bd-a352-3db8329d003b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:16.385436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:16.415760 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:16.416415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:16.416690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:16.416962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:16.428816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:16.430273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:16.447495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:16.448713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:16.449821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:16.449821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:16.449821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:16.451613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 469/940] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:16.461416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473eeb10-b222-41ef-a5b2-3b81e2d28321 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:16.494548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:16.589695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473eeb10-b222-41ef-a5b2-3b81e2d28321 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:16.626255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:16.626897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:16.627130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:16.627345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:16.627996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:16.628369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:16.629753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:16.643274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 472/941] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:16.648795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:16.763822 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c419cb3-0b3b-4838-bfe0-539b35171e9f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:16.810033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f64584-e737-4c5c-b68e-cd0e305d00c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:16.810675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f64584-e737-4c5c-b68e-cd0e305d00c3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:16.811773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f64584-e737-4c5c-b68e-cd0e305d00c3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:16.812163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f64584-e737-4c5c-b68e-cd0e305d00c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:16.830205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f64584-e737-4c5c-b68e-cd0e305d00c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:16.833011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f64584-e737-4c5c-b68e-cd0e305d00c3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-c9f64584-e737-4c5c-b68e-cd0e305d00c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:16.857147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:16.858865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:16.859778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:16.859778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:16.859778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:16.859778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:16.859778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:16.859778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:16.859778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:16.859778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:16.860679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 470/942] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:16.866680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:16.930563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeae582-257b-4b26-b537-352596188255 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:16.930784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeae582-257b-4b26-b537-352596188255 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:16.931080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeae582-257b-4b26-b537-352596188255 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:16.932579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 471/943] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:16.938362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03722df2-c85a-4c7d-8452-99fe9924ad1a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:16.938669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03722df2-c85a-4c7d-8452-99fe9924ad1a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:16.938947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03722df2-c85a-4c7d-8452-99fe9924ad1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:16.963105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:16.971667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-03722df2-c85a-4c7d-8452-99fe9924ad1a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:16.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-03722df2-c85a-4c7d-8452-99fe9924ad1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:17.002153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc2b02f-4cb8-4b24-9eaf-32ed816fc797 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:17.024080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:17.055674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:17.056690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:17.057218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:17.057694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:17.058622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:17.059159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:17.061718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:17.085258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 473/944] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:28:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:17.088136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:17.127058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 472/945] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:16 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:17.216753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5335725a-27e2-48d7-adff-865cfd9a8124 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:17.259197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158ee217-9947-4f98-b446-f2f3b6db4065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:17.260310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ee217-9947-4f98-b446-f2f3b6db4065 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:17.260844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ee217-9947-4f98-b446-f2f3b6db4065 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:17.261337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ee217-9947-4f98-b446-f2f3b6db4065 None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/18211b2dc99f43769685855f07c64a3e/roles/f772879e7c8a4395a3cdc078d84cc832` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:17.265130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158ee217-9947-4f98-b446-f2f3b6db4065 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=18211b2dc99f43769685855f07c64a3e, role_id=f772879e7c8a4395a3cdc078d84cc832)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:17.268053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158ee217-9947-4f98-b446-f2f3b6db4065 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:17.283644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:17.289791 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 474/946] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:28:17 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/18211b2dc99f43769685855f07c64a3e/roles/f772879e7c8a4395a3cdc078d84cc832 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:28:17.454325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03722df2-c85a-4c7d-8452-99fe9924ad1a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:17.482042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d8682f-c86a-4c21-9a7a-9609ea6b17e7 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:17.482359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d8682f-c86a-4c21-9a7a-9609ea6b17e7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:17.482599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d8682f-c86a-4c21-9a7a-9609ea6b17e7 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:17.484848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 475/947] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:17.490553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff14571-ded8-4bd3-84f3-044f49cab6ae None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:17.490965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff14571-ded8-4bd3-84f3-044f49cab6ae None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:17.491206 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff14571-ded8-4bd3-84f3-044f49cab6ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:17.522827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2ff14571-ded8-4bd3-84f3-044f49cab6ae None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:17.524574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2ff14571-ded8-4bd3-84f3-044f49cab6ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:17.638953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03722df2-c85a-4c7d-8452-99fe9924ad1a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:17.670940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 476/948] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:17.707792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:17.709071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:17.709848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:17.710616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] PATH_INFO: `/v3/services/object-store` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:17.711991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:17.714683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:17.737397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:17.739315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:17.741547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:17.751057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 473/949] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:17 2022] GET /identity/v3/services/object-store => generated 93 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:17.757325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff14571-ded8-4bd3-84f3-044f49cab6ae None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:17.909842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff14571-ded8-4bd3-84f3-044f49cab6ae None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:17.959200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:17.963326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:17.964127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:17.964566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:17.964840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:17.965719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:17.966151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:17.968117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:17.975578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 477/950] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:17 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:17.985138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:18.027861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:18.065789 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:18.150626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514a0692-33d7-4095-b8a3-a4f2bbb249bf None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:18.205383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:18.206373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:18.206837 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:18.207244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:18.208139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:18.208499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:18.209896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:18.222764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 474/951] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:17 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:18.226992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:18.261583 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c27b0-8c52-4694-a941-91b395b16ba3 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:18.318872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:18.320494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:18.321502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:18.322711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:18.324079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:18.326933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:18.331432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING keystone.server.flask.common [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 29 22:28:18.360448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.sql.core [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 29 22:28:18.360448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 29 22:28:18.360448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [parameters: {'id': '78ec297dc76c4fed99637a87e80fccb6', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 29 22:28:18.360448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=103002) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 29 22:28:18.365838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 78ec297dc76c4fed99637a87e80fccb6.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 78ec297dc76c4fed99637a87e80fccb6. Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application conn.query(q) Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 22:28:18.376458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result.read() Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 29 22:28:18.377447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application next(self.gen) Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application yield resource Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application next(self.gen) Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.session.rollback() Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.force_reraise() Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise self.value Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application session.commit() Aug 29 22:28:18.378467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._prepare_impl() Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.session.flush() Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._flush(objects) Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application compat.raise_( Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application flush_context.execute() Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 29 22:28:18.379410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rec.execute(self) Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 29 22:28:18.380394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application conn.query(q) Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result.read() Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 22:28:18.381211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [parameters: {'id': '78ec297dc76c4fed99637a87e80fccb6', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:18.382081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:18.384434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 78ec297dc76c4fed99637a87e80fccb6. Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 478/952] 10.222.0.77 () {60 vars in 1140 bytes} [Mon Aug 29 22:28:18 2022] POST /identity/v3/projects => generated 262 bytes in 315 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:18.387316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:18.389863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b9c734-1c89-4acf-ae84-312e419a09f2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:18.432101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f99030-fbf5-494c-a86a-be63962a9c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:18.432744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f99030-fbf5-494c-a86a-be63962a9c47 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:18.433022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f99030-fbf5-494c-a86a-be63962a9c47 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:18.433289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f99030-fbf5-494c-a86a-be63962a9c47 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:18.434178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f99030-fbf5-494c-a86a-be63962a9c47 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:18.435946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f99030-fbf5-494c-a86a-be63962a9c47 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:18.459781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:18.470771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 475/953] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:28:18 2022] POST /identity/v3/endpoints => generated 349 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:28:18.554066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fbe9e4-41e5-43a7-b364-f601acc33bb0 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:18.554385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fbe9e4-41e5-43a7-b364-f601acc33bb0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:18.554671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fbe9e4-41e5-43a7-b364-f601acc33bb0 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:18.556054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 476/954] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:18.560865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac351ed0-97ca-43f4-9234-ed9855093130 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:18.563513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaa7726-0ebd-4115-a503-26d84b5a30e9 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:18.564009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaa7726-0ebd-4115-a503-26d84b5a30e9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:18.564627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaa7726-0ebd-4115-a503-26d84b5a30e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:18.591789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-bdaa7726-0ebd-4115-a503-26d84b5a30e9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:18.593877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-bdaa7726-0ebd-4115-a503-26d84b5a30e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:18.649141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:18.649827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:18.651351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:18.651351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:18.664070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:18.673358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:18.690428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:18.691524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:18.692559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 479/955] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:18.696993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:18.706648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 477/956] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:18 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:18.823600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec1a0bf-de65-427f-8f6f-1ae2deb9957e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:18.832917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:18.870769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:18.871482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:18.871966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:18.872425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:18.873298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] RBAC: Adding query filter params (name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:18.873805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:18.875666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:18.897190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 480/957] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:28:18 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:19.016530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaa7726-0ebd-4115-a503-26d84b5a30e9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:19.134553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:19.171299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaa7726-0ebd-4115-a503-26d84b5a30e9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:19.237387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:19.238410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:19.238931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:19.239532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] PATH_INFO: `/v3/services/compute` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:19.240332 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:19.241943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:19.253118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:19.254238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:19.255417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:19.258381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 478/958] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:19 2022] GET /identity/v3/services/compute => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:19.264203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:19.402283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ad5b9a-2aee-4212-8506-61010b9182ea None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:19.450109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab15ea55-fe5f-4e5b-9763-f3703ea3a200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:19.450749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab15ea55-fe5f-4e5b-9763-f3703ea3a200 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:19.451040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab15ea55-fe5f-4e5b-9763-f3703ea3a200 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:19.451310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab15ea55-fe5f-4e5b-9763-f3703ea3a200 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:19.452645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab15ea55-fe5f-4e5b-9763-f3703ea3a200 None admin] RBAC: Adding query filter params (name=compute) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:19.453239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab15ea55-fe5f-4e5b-9763-f3703ea3a200 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:19.456112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab15ea55-fe5f-4e5b-9763-f3703ea3a200 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:19.464767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 481/959] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:28:19 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:19.470884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:19.591390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfe4fd8-b03c-4fac-97a1-bd3d30b5653b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:19.629164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c42022-2a6c-4800-b23e-162292b490ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:19.629773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c42022-2a6c-4800-b23e-162292b490ab None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:19.630048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c42022-2a6c-4800-b23e-162292b490ab None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:19.630315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c42022-2a6c-4800-b23e-162292b490ab None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:19.630991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c42022-2a6c-4800-b23e-162292b490ab None admin] RBAC: Adding query filter params (type=compute) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:19.631294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c42022-2a6c-4800-b23e-162292b490ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:19.632665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c42022-2a6c-4800-b23e-162292b490ab None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:19.637994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 479/960] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:28:19 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:19.691845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac49c637-60e1-4d8f-af1b-1089172a0005 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:19.692355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac49c637-60e1-4d8f-af1b-1089172a0005 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:19.692355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac49c637-60e1-4d8f-af1b-1089172a0005 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:19.694094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 482/961] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:19.700309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37544fe-76d7-419e-8bbe-27f4c1966417 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:19.700614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37544fe-76d7-419e-8bbe-27f4c1966417 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:19.700857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37544fe-76d7-419e-8bbe-27f4c1966417 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:19.709805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:19.723704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b37544fe-76d7-419e-8bbe-27f4c1966417 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:19.725218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b37544fe-76d7-419e-8bbe-27f4c1966417 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:19.814434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 480/962] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:20.097986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac49c637-60e1-4d8f-af1b-1089172a0005 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:20.135174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:20.246084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac49c637-60e1-4d8f-af1b-1089172a0005 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:20.316118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8e6a14-19d9-4464-b611-2ca4e6420a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:20.317452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8e6a14-19d9-4464-b611-2ca4e6420a13 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:20.318034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8e6a14-19d9-4464-b611-2ca4e6420a13 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:20.318526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8e6a14-19d9-4464-b611-2ca4e6420a13 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:20.339113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8e6a14-19d9-4464-b611-2ca4e6420a13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-af8e6a14-19d9-4464-b611-2ca4e6420a13 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:20.349469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:20.350414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:20.351332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:20.351995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 483/963] 10.222.0.77 () {56 vars in 1120 bytes} [Mon Aug 29 22:28:20 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:20.362369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37544fe-76d7-419e-8bbe-27f4c1966417 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:20.551295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37544fe-76d7-419e-8bbe-27f4c1966417 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:20.591838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:20.592997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:20.593283 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:20.593550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:20.594227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:20.594533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:20.595954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:20.597595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:20.605789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 481/964] 10.222.0.77 () {56 vars in 1115 bytes} [Mon Aug 29 22:28:20 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:20.611866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8e6a14-19d9-4464-b611-2ca4e6420a13 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:20.719444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:20.729071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8e6a14-19d9-4464-b611-2ca4e6420a13 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:20.778047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09299d4-2967-44f1-a9d3-60144b30e535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:20.778725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09299d4-2967-44f1-a9d3-60144b30e535 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:20.779028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09299d4-2967-44f1-a9d3-60144b30e535 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:20.779302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09299d4-2967-44f1-a9d3-60144b30e535 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:20.799305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09299d4-2967-44f1-a9d3-60144b30e535 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:20.800778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09299d4-2967-44f1-a9d3-60144b30e535 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-b09299d4-2967-44f1-a9d3-60144b30e535 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:20.818076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:20.819142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:20.820063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:20.820063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:20.820063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:20.820063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 484/965] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:20.825262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:20.898766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c344048a-1ae7-412a-bee0-43fcc13d3a0a None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:20.899078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c344048a-1ae7-412a-bee0-43fcc13d3a0a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:20.899300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c344048a-1ae7-412a-bee0-43fcc13d3a0a None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:20.900905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 485/966] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:20.906998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4aded0-c563-43bf-9fb0-480953b2477b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:20.907302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4aded0-c563-43bf-9fb0-480953b2477b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:20.907562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4aded0-c563-43bf-9fb0-480953b2477b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:20.928835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-da4aded0-c563-43bf-9fb0-480953b2477b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:20.930471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-da4aded0-c563-43bf-9fb0-480953b2477b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:21.021947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9f639-398e-45ea-a2a2-b733370503e4 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:21.022652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 486/967] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:21.081226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4af15e-c58b-432b-826e-80c3698b9da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:21.081853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4af15e-c58b-432b-826e-80c3698b9da5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:21.082210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4af15e-c58b-432b-826e-80c3698b9da5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:21.082506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4af15e-c58b-432b-826e-80c3698b9da5 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:21.083294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4af15e-c58b-432b-826e-80c3698b9da5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:21.083674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4af15e-c58b-432b-826e-80c3698b9da5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:21.085761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4af15e-c58b-432b-826e-80c3698b9da5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:21.104882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 482/968] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:21.110104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4aded0-c563-43bf-9fb0-480953b2477b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:21.191134 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:21.200767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24601e9c-48cb-4243-bc3e-2ac2423ade99 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:21.201058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24601e9c-48cb-4243-bc3e-2ac2423ade99 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:21.201390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24601e9c-48cb-4243-bc3e-2ac2423ade99 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:21.203622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 483/969] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:21.209020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9096b-2764-41c8-9780-b28f140534d3 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:21.209336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9096b-2764-41c8-9780-b28f140534d3 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:21.209612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9096b-2764-41c8-9780-b28f140534d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:21.232940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-37e9096b-2764-41c8-9780-b28f140534d3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:21.233988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4aded0-c563-43bf-9fb0-480953b2477b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:21.234260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-37e9096b-2764-41c8-9780-b28f140534d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:21.293133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:21.294130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:21.294629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:21.295169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:21.310596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:21.314408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:21.318839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:21.346783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 484/970] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:21.347264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:21.348516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:21.349473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 487/971] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:28:21 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:21.358079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e9096b-2764-41c8-9780-b28f140534d3 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:21.376819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:21.518683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e9096b-2764-41c8-9780-b28f140534d3 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:21.570631 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaaa434-f296-4e27-ab46-7e8b5b17fe86 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:21.571032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:21.571891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:21.572372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:21.572847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:21.574393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:21.574627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:21.576676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:21.606220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 485/972] 10.222.0.77 () {56 vars in 1149 bytes} [Mon Aug 29 22:28:21 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:21.607967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:21.612020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:21.618971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:21.619956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:21.620439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:21.620887 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:21.621670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:21.623333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:21.635890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:21.636975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:21.637947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:21.647932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 488/973] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:21.652242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:21.668585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:21.746326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee8c13-fc1a-4b00-ab3c-638452299d1f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:21.801559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a18978-c3ca-460a-91c4-83d9e54fc6c8 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:21.810162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f762dc48-f68e-436c-a479-2eebb84908d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:21.810791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f762dc48-f68e-436c-a479-2eebb84908d3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:21.812881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f762dc48-f68e-436c-a479-2eebb84908d3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:21.812881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f762dc48-f68e-436c-a479-2eebb84908d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:21.859265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:21.860283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:21.860692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:21.861100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:21.861885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:21.862316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:21.864174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:21.881706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 489/974] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:21 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:21.881706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:21.928274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f762dc48-f68e-436c-a479-2eebb84908d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:21.929742 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f762dc48-f68e-436c-a479-2eebb84908d3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:21.962661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:22.011039 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:22.027468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee41e63-3b50-4c2d-877f-d87cc8b4ba5a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:22.083571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:22.084375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.084771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.085202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.086292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:22.088761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:22.114361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.sql.core [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 29 22:28:22.114361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 29 22:28:22.114361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [parameters: {'user_id': '710d49f8906c4434a9544322669ab3b0', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 29 22:28:22.114361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=103002) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 29 22:28:22.115114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 486/975] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:21 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 502 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 22:28:22.115114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f762dc48-f68e-436c-a479-2eebb84908d3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application conn.query(q) Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result.read() Aug 29 22:28:22.116946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 22:28:22.119229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application next(self.gen) Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application yield resource Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application next(self.gen) Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.session.rollback() Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.force_reraise() Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise self.value Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application session.commit() Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 29 22:28:22.124192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._prepare_impl() Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.session.flush() Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._flush(objects) Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application compat.raise_( Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application flush_context.execute() Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rec.execute(self) Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 29 22:28:22.126344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 29 22:28:22.127601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result = self._query(query) Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application conn.query(q) Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application result.read() Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 29 22:28:22.128771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [parameters: {'user_id': '710d49f8906c4434a9544322669ab3b0', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:22.130724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 29 22:28:22.133412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 29 22:28:22.135900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 29 22:28:22.135900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 29 22:28:22.135900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 29 22:28:22.135900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 29 22:28:22.135900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 29 22:28:22.135900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:22.135900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 490/976] 10.222.0.77 () {60 vars in 1131 bytes} [Mon Aug 29 22:28:21 2022] POST /identity/v3/users => generated 160 bytes in 240 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:22.135900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:22.246263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f762dc48-f68e-436c-a479-2eebb84908d3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:22.251648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a44d6-b7a6-47c3-a1e4-4d1ac9c88384 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:22.283853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:22.285047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:22.285765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.286525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.286971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.287989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:22.288511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:22.290080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:22.302012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb9b392-f2e2-4cf3-b7ad-8116967bdfd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:22.302708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb9b392-f2e2-4cf3-b7ad-8116967bdfd6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.303072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb9b392-f2e2-4cf3-b7ad-8116967bdfd6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.303400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 487/977] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:22 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:22.303511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb9b392-f2e2-4cf3-b7ad-8116967bdfd6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.304565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:22.321486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb9b392-f2e2-4cf3-b7ad-8116967bdfd6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:22.322899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:22.323326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb9b392-f2e2-4cf3-b7ad-8116967bdfd6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-ccb9b392-f2e2-4cf3-b7ad-8116967bdfd6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:22.351683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:22.353141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:22.354434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:22.354434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:22.354434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:22.354434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 491/978] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:22.354757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a08bb-797c-46d0-b024-1d639ada016d None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.354997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a08bb-797c-46d0-b024-1d639ada016d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.355265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a08bb-797c-46d0-b024-1d639ada016d None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.360197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 492/979] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:22.361038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105a08bb-797c-46d0-b024-1d639ada016d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:22.454581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69301e0-bc9f-4a6c-8b62-9fa0c8c05c1b None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:22.506242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105a08bb-797c-46d0-b024-1d639ada016d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:22.519877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:22.520947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.521689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.521889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.534944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:22.537742 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:22.558212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:22.560124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 488/980] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:28:22 2022] GET /identity/v3/users/cinder => generated 84 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:22.561994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:22.573445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b87146-8068-44a1-9d74-0707302352d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:22.574412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b87146-8068-44a1-9d74-0707302352d4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.574917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b87146-8068-44a1-9d74-0707302352d4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.575254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b87146-8068-44a1-9d74-0707302352d4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.576371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b87146-8068-44a1-9d74-0707302352d4 None admin] RBAC: Adding query filter params (interface=admin, service_id=4214798a2d2d463b989437959295ddd2, region_id=RegionOne) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:22.577399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b87146-8068-44a1-9d74-0707302352d4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:22.579666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b87146-8068-44a1-9d74-0707302352d4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:22.593317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 493/981] 10.222.0.77 () {56 vars in 1249 bytes} [Mon Aug 29 22:28:22 2022] GET /identity/v3/endpoints?service_id=4214798a2d2d463b989437959295ddd2&interface=admin®ion_id=RegionOne => generated 193 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:22.593317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e21096-05b1-46c9-8767-63c4f90b79aa None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.593317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e21096-05b1-46c9-8767-63c4f90b79aa None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.593317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e21096-05b1-46c9-8767-63c4f90b79aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.617800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-41e21096-05b1-46c9-8767-63c4f90b79aa None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:22.619696 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-41e21096-05b1-46c9-8767-63c4f90b79aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:22.649451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:22.657272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:22.695675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 494/982] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:22.696371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e21096-05b1-46c9-8767-63c4f90b79aa None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:22.704343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35a17af-82bd-48bb-bbb3-b858bb9055d5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:22.758260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:22.758920 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.759270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.759446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.760099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:22.760447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:22.761782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:22.783204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 489/983] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:22.784393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:22.858526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e21096-05b1-46c9-8767-63c4f90b79aa None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:22.926315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:22.926888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:22.927185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:22.927461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:22.928134 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] RBAC: Adding query filter params (name=cinder) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:22.928439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:22.929888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:22.971298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61119f65-8ddd-44f6-99fa-1633034ede09 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:22.986777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 495/984] 10.222.0.77 () {56 vars in 1101 bytes} [Mon Aug 29 22:28:22 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:22.986777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:23.017437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:23.026760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:23.036791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9150fb1-da8f-4f5b-adc1-8267a19c1b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:23.037998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9150fb1-da8f-4f5b-adc1-8267a19c1b94 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:23.038457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9150fb1-da8f-4f5b-adc1-8267a19c1b94 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:23.038925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9150fb1-da8f-4f5b-adc1-8267a19c1b94 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:23.039945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9150fb1-da8f-4f5b-adc1-8267a19c1b94 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:23.043821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9150fb1-da8f-4f5b-adc1-8267a19c1b94 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:23.058779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 490/985] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:28:22 2022] GET /identity/v3/services => generated 1497 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:28:23.058779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9150fb1-da8f-4f5b-adc1-8267a19c1b94 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:23.193518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9833d9a5-2d67-4e00-98d7-d2b0c90bbacf None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:23.268734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9150fb1-da8f-4f5b-adc1-8267a19c1b94 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:23.269026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:23.269977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:23.270579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:23.271242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:23.284907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:23.287197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:23.311958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:23.313989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:23.316031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 496/986] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:23.324394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:23.345208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b1ad3-fbdb-49bb-bcf4-3e79aff1b944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:23.346122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b1ad3-fbdb-49bb-bcf4-3e79aff1b944 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:23.346619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b1ad3-fbdb-49bb-bcf4-3e79aff1b944 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:23.347111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b1ad3-fbdb-49bb-bcf4-3e79aff1b944 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:23.348288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2b1ad3-fbdb-49bb-bcf4-3e79aff1b944 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:23.350840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2b1ad3-fbdb-49bb-bcf4-3e79aff1b944 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:23.376605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 491/987] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:28:23 2022] POST /identity/v3/services => generated 242 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:23.495840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:23.525238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0cee96-228d-4d32-b18b-2a2b5aad28f3 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:23.601540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c798b4-a974-4377-98b5-e167b7d9ac60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:23.602726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c798b4-a974-4377-98b5-e167b7d9ac60 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:23.603333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c798b4-a974-4377-98b5-e167b7d9ac60 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:23.603890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c798b4-a974-4377-98b5-e167b7d9ac60 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:23.605298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c798b4-a974-4377-98b5-e167b7d9ac60 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:23.605930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c798b4-a974-4377-98b5-e167b7d9ac60 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:23.609467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c798b4-a974-4377-98b5-e167b7d9ac60 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:23.628367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 497/988] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:28:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:23.638553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b1ad3-fbdb-49bb-bcf4-3e79aff1b944 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:23.762051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:23.868868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b1ad3-fbdb-49bb-bcf4-3e79aff1b944 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:23.936157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b1bd97-87f6-4bb9-87ca-b909c9460aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:23.937807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b1bd97-87f6-4bb9-87ca-b909c9460aef None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:23.938361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b1bd97-87f6-4bb9-87ca-b909c9460aef None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:23.938938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b1bd97-87f6-4bb9-87ca-b909c9460aef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:23.940437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b1bd97-87f6-4bb9-87ca-b909c9460aef None admin] RBAC: Adding query filter params (role.id=f772879e7c8a4395a3cdc078d84cc832, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=18211b2dc99f43769685855f07c64a3e) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:23.941536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b1bd97-87f6-4bb9-87ca-b909c9460aef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:23.944333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b1bd97-87f6-4bb9-87ca-b909c9460aef None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:23.959136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 492/989] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:23 2022] GET /identity/v3/role_assignments?user.id=18211b2dc99f43769685855f07c64a3e&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=f772879e7c8a4395a3cdc078d84cc832 => generated 608 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:25.475107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b215463-c7dc-4af3-927d-f4fcf74ae52c None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:25.475926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b215463-c7dc-4af3-927d-f4fcf74ae52c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:25.475926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b215463-c7dc-4af3-927d-f4fcf74ae52c None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:25.478112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 498/990] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:25.488597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4b7b5-df00-4928-b24c-4fabb25213f3 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:25.488597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4b7b5-df00-4928-b24c-4fabb25213f3 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:25.488597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4b7b5-df00-4928-b24c-4fabb25213f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:25.518939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-10d4b7b5-df00-4928-b24c-4fabb25213f3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:25.520908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-10d4b7b5-df00-4928-b24c-4fabb25213f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:25.649522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 493/991] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:25 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:25.976076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f9c49d-2c13-49e7-9706-1fd2a21ac66c None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:25.976571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f9c49d-2c13-49e7-9706-1fd2a21ac66c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:25.977036 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f9c49d-2c13-49e7-9706-1fd2a21ac66c None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:25.979902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 499/992] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:25 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:25.989605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd053677-4a44-42a1-bd99-b532cf8d7149 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:25.990114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd053677-4a44-42a1-bd99-b532cf8d7149 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:25.990607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd053677-4a44-42a1-bd99-b532cf8d7149 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:26.021798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-dd053677-4a44-42a1-bd99-b532cf8d7149 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:26.026084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-dd053677-4a44-42a1-bd99-b532cf8d7149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:26.105363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:26.135511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f9c49d-2c13-49e7-9706-1fd2a21ac66c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:26.148422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 494/993] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:25 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:26.299991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f9c49d-2c13-49e7-9706-1fd2a21ac66c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:26.345847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:26.346524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:26.346843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:26.347121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:26.347734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:26.349223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:26.360490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:26.362489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 29 22:28:26.363813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:26.366393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 500/994] 10.222.0.77 () {56 vars in 1126 bytes} [Mon Aug 29 22:28:26 2022] GET /identity/v3/services/object-store => generated 93 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:26.367762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd053677-4a44-42a1-bd99-b532cf8d7149 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:26.473223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:26.502325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:26.543694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd053677-4a44-42a1-bd99-b532cf8d7149 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:26.622431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b196b4c3-e2ba-4e42-8e12-adbecd54308c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:26.623311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b196b4c3-e2ba-4e42-8e12-adbecd54308c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:26.623643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b196b4c3-e2ba-4e42-8e12-adbecd54308c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:26.624003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b196b4c3-e2ba-4e42-8e12-adbecd54308c None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:26.624819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b196b4c3-e2ba-4e42-8e12-adbecd54308c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:26.625167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b196b4c3-e2ba-4e42-8e12-adbecd54308c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:26.626688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b196b4c3-e2ba-4e42-8e12-adbecd54308c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:26.632894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 495/995] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:26 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:26.634331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7404bac-03c2-4a30-9be0-c67451843012 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:26.634798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7404bac-03c2-4a30-9be0-c67451843012 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:26.635246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7404bac-03c2-4a30-9be0-c67451843012 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:26.637580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 496/996] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:26.643285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7404bac-03c2-4a30-9be0-c67451843012 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:26.690073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ec58ac-44ae-4f7d-80ec-2eaf2fc8651f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:26.712500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:26.752829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b37241-5830-424b-b892-59548fd2162a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:26.753896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b37241-5830-424b-b892-59548fd2162a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:26.754241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b37241-5830-424b-b892-59548fd2162a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:26.754808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b37241-5830-424b-b892-59548fd2162a None admin] PATH_INFO: `/v3/services/compute` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:26.755861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b37241-5830-424b-b892-59548fd2162a None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:26.758253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b37241-5830-424b-b892-59548fd2162a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-89b37241-5830-424b-b892-59548fd2162a None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:26.774419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:26.777181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:26.779167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 501/997] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:26 2022] GET /identity/v3/services/compute => generated 88 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:26.781447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b29e229-cbe0-43b6-b5f3-70e3fab1f091 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:26.781447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b29e229-cbe0-43b6-b5f3-70e3fab1f091 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:26.781447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b29e229-cbe0-43b6-b5f3-70e3fab1f091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:26.806819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8b29e229-cbe0-43b6-b5f3-70e3fab1f091 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:26.807825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8b29e229-cbe0-43b6-b5f3-70e3fab1f091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:26.837722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7404bac-03c2-4a30-9be0-c67451843012 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:26.894252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:26.895297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:26.895812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:26.896314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:26.897551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:26.898118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:26.900914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:26.914784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 497/998] 10.222.0.77 () {56 vars in 1122 bytes} [Mon Aug 29 22:28:26 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:26.914784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:26.916866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:26.924348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 502/999] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:26 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:26.925876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b29e229-cbe0-43b6-b5f3-70e3fab1f091 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:27.078946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0cbc79-b305-48ef-a9e3-f442e49ce24e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:27.084354 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:27.120072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b29e229-cbe0-43b6-b5f3-70e3fab1f091 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:27.131799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:27.132587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:27.133005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:27.133426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:27.134241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] RBAC: Adding query filter params (name=compute) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:27.134687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:27.136657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:27.143182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 498/1000] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:28:26 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:27.149838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:27.189379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b8fdca-6afe-4c08-b0e4-8bae7f2d196d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:27.190515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b8fdca-6afe-4c08-b0e4-8bae7f2d196d None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:27.191038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b8fdca-6afe-4c08-b0e4-8bae7f2d196d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:27.191512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b8fdca-6afe-4c08-b0e4-8bae7f2d196d None admin] PATH_INFO: `/v3/endpoints` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:27.192835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b8fdca-6afe-4c08-b0e4-8bae7f2d196d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:27.195824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b8fdca-6afe-4c08-b0e4-8bae7f2d196d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:27.224992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 503/1001] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:28:26 2022] POST /identity/v3/endpoints => generated 325 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:27.226565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b8fdca-6afe-4c08-b0e4-8bae7f2d196d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:27.338258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb7a55-f623-494f-8ab4-a58d26186c06 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:27.354603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:27.420365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:27.421766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:27.422222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:27.422599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:27.423536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] RBAC: Adding query filter params (type=compute) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:27.423953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:27.425721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:27.438768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b8fdca-6afe-4c08-b0e4-8bae7f2d196d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:27.440184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 499/1002] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:28:27 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:27.441507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:27.510872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:27.533595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:27.535388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:27.536050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:27.536961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] PATH_INFO: `/v3/services/image` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:27.538012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:27.540675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:27.559797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:27.562095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:27.564759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 504/1003] 10.222.0.77 () {56 vars in 1105 bytes} [Mon Aug 29 22:28:27 2022] GET /identity/v3/services/image => generated 86 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:27.571251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:27.683820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95ebc14-5896-4812-af18-5c50314bc6bd None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:27.751728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:27.752709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:27.753257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:27.753779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:27.755065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] RBAC: Adding query filter params (interface=public, service_id=73231b6be5de4780ac29262dae50eee9, region_id=RegionOne) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:27.755943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:27.757710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afb8264-6cd9-4bc2-a9dc-6a2a45a4c099 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:27.758616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:27.775598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 500/1004] 10.222.0.77 () {56 vars in 1251 bytes} [Mon Aug 29 22:28:27 2022] GET /identity/v3/endpoints?service_id=73231b6be5de4780ac29262dae50eee9&interface=public®ion_id=RegionOne => generated 194 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:27.777172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:27.813627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:27.852448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:27.853899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:27.854661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:27.855500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:27.857064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] RBAC: Adding query filter params (name=image) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:27.857989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:27.860104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:27.874882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 505/1005] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:27 2022] GET /identity/v3/services?name=image => generated 122 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:27.880136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:28.008423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:28.032116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c309e63b-bd04-483d-be43-15d26b34e899 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:28.105459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ffe840-4d11-4c53-8451-8d6c5e5a838e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:28.106529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ffe840-4d11-4c53-8451-8d6c5e5a838e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:28.107053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ffe840-4d11-4c53-8451-8d6c5e5a838e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:28.107555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ffe840-4d11-4c53-8451-8d6c5e5a838e None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:28.109244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ffe840-4d11-4c53-8451-8d6c5e5a838e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:28.111912 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ffe840-4d11-4c53-8451-8d6c5e5a838e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:28.125808 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 501/1006] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:28:27 2022] GET /identity/v3/services => generated 1727 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:28:28.126064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78b92ba-27a8-4328-9842-0b9e84eadda7 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:28.197047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77da63c-794a-4200-a580-e829ca7abce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:28.198132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77da63c-794a-4200-a580-e829ca7abce4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:28.198626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77da63c-794a-4200-a580-e829ca7abce4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:28.199232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77da63c-794a-4200-a580-e829ca7abce4 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:28.200533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77da63c-794a-4200-a580-e829ca7abce4 None admin] RBAC: Adding query filter params (type=image) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:28.201073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77da63c-794a-4200-a580-e829ca7abce4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:28.203912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77da63c-794a-4200-a580-e829ca7abce4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:28.213075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 506/1007] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:27 2022] GET /identity/v3/services?type=image => generated 122 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:29.936252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379d8c8f-dbb6-4135-b868-ff88ba7ce4cc None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:29.937017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379d8c8f-dbb6-4135-b868-ff88ba7ce4cc None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:29.937449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379d8c8f-dbb6-4135-b868-ff88ba7ce4cc None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:29.942797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 502/1008] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:29 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:29.948566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6362ac21-7ea9-4c8e-ae70-b3c95ace516a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:29.949318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6362ac21-7ea9-4c8e-ae70-b3c95ace516a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:29.949606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6362ac21-7ea9-4c8e-ae70-b3c95ace516a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:29.981241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-6362ac21-7ea9-4c8e-ae70-b3c95ace516a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:29.983552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6362ac21-7ea9-4c8e-ae70-b3c95ace516a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:30.069921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:30.109754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 507/1009] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:29 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:30.402470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379d8c8f-dbb6-4135-b868-ff88ba7ce4cc None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:30.580282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379d8c8f-dbb6-4135-b868-ff88ba7ce4cc None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:30.664497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b2d4b4-cb2b-4b9a-8e4e-cd690d695fd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:30.665988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b2d4b4-cb2b-4b9a-8e4e-cd690d695fd0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:30.666713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b2d4b4-cb2b-4b9a-8e4e-cd690d695fd0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:30.667444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b2d4b4-cb2b-4b9a-8e4e-cd690d695fd0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:30.669672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b2d4b4-cb2b-4b9a-8e4e-cd690d695fd0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:30.672825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b2d4b4-cb2b-4b9a-8e4e-cd690d695fd0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:30.676266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 503/1010] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:30.685950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6362ac21-7ea9-4c8e-ae70-b3c95ace516a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:30.874050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6362ac21-7ea9-4c8e-ae70-b3c95ace516a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:30.936585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394330bf-175b-4a8e-bc62-df1ea77fdad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:30.938030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394330bf-175b-4a8e-bc62-df1ea77fdad5 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:30.938482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394330bf-175b-4a8e-bc62-df1ea77fdad5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:30.938961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394330bf-175b-4a8e-bc62-df1ea77fdad5 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:30.940054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394330bf-175b-4a8e-bc62-df1ea77fdad5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:30.942500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394330bf-175b-4a8e-bc62-df1ea77fdad5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:30.985081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef809a98-f215-4030-b75d-3e1b295c221b None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:30.985928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef809a98-f215-4030-b75d-3e1b295c221b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:30.986088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef809a98-f215-4030-b75d-3e1b295c221b None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:30.988854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 504/1011] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:30.994362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 508/1012] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:28:30 2022] POST /identity/v3/projects => generated 311 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:30.997744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f49b0-0549-4433-923d-2faf169b56c3 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:30.998254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f49b0-0549-4433-923d-2faf169b56c3 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:30.998774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f49b0-0549-4433-923d-2faf169b56c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:31.034042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-746f49b0-0549-4433-923d-2faf169b56c3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:31.036223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-746f49b0-0549-4433-923d-2faf169b56c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:31.076002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:31.148255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784791b-da6d-451c-925c-9a78289128ff None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:31.148767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784791b-da6d-451c-925c-9a78289128ff None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:31.148767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784791b-da6d-451c-925c-9a78289128ff None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:31.151981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 509/1013] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:31.160358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3c987b-e674-4def-9e62-ebf629827b18 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:31.160814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3c987b-e674-4def-9e62-ebf629827b18 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:31.161226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3c987b-e674-4def-9e62-ebf629827b18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:31.192427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2d3c987b-e674-4def-9e62-ebf629827b18 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:31.194255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2d3c987b-e674-4def-9e62-ebf629827b18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:31.195234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 505/1014] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:30 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:31.236987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:31.300624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 510/1015] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:31 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:31.646892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746f49b0-0549-4433-923d-2faf169b56c3 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:31.715730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3c987b-e674-4def-9e62-ebf629827b18 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:31.835462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746f49b0-0549-4433-923d-2faf169b56c3 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:31.902519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:31.903189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:31.903473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:31.903740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] PATH_INFO: `/v3/services/compute` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:31.904471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:31.906290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:31.922164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:31.924457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 29 22:28:31.926183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:31.927864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3c987b-e674-4def-9e62-ebf629827b18 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:31.930999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 506/1016] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:31 2022] GET /identity/v3/services/compute => generated 88 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:31.932271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:31.967257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:32.006594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab2094a-76bb-4e12-8c11-2bf8398acc9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:32.007676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab2094a-76bb-4e12-8c11-2bf8398acc9b None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:32.008179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab2094a-76bb-4e12-8c11-2bf8398acc9b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:32.008664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab2094a-76bb-4e12-8c11-2bf8398acc9b None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:32.009894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab2094a-76bb-4e12-8c11-2bf8398acc9b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:32.012462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab2094a-76bb-4e12-8c11-2bf8398acc9b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:32.038772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 511/1017] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:28:31 2022] POST /identity/v3/services => generated 242 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:32.130432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466ed274-bece-4eb2-8a97-2123cbbe4283 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:32.197917 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:32.199379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:32.200165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:32.200914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:32.202062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] RBAC: Adding query filter params (name=compute) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:32.203210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:32.205865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:32.216996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 507/1018] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:28:31 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:32.223293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab2094a-76bb-4e12-8c11-2bf8398acc9b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:32.258997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:32.377651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab2094a-76bb-4e12-8c11-2bf8398acc9b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:32.444250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6c2687-5b45-47e6-a02f-a32372e21ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:32.445302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6c2687-5b45-47e6-a02f-a32372e21ee3 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:32.445795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6c2687-5b45-47e6-a02f-a32372e21ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:32.446273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6c2687-5b45-47e6-a02f-a32372e21ee3 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:32.447484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6c2687-5b45-47e6-a02f-a32372e21ee3 None admin] RBAC: Adding query filter params (type=compute) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:32.448096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6c2687-5b45-47e6-a02f-a32372e21ee3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:32.450684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6c2687-5b45-47e6-a02f-a32372e21ee3 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:32.460646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 512/1019] 10.222.0.77 () {56 vars in 1112 bytes} [Mon Aug 29 22:28:32 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:32.470971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:32.631627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:32.678079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15321305-cae9-498e-9c61-14a11625b87a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:32.740862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba4bec0-ef27-485e-8d93-0ab3bb311b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:32.741669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4bec0-ef27-485e-8d93-0ab3bb311b25 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:32.742087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4bec0-ef27-485e-8d93-0ab3bb311b25 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:32.742506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4bec0-ef27-485e-8d93-0ab3bb311b25 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:32.743388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba4bec0-ef27-485e-8d93-0ab3bb311b25 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:32.744982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba4bec0-ef27-485e-8d93-0ab3bb311b25 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:32.773924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 508/1020] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:28:32 2022] POST /identity/v3/endpoints => generated 334 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:33.811921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a68a94-b5f0-4197-9bb0-e5fc1132f528 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:33.812700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a68a94-b5f0-4197-9bb0-e5fc1132f528 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:33.812700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a68a94-b5f0-4197-9bb0-e5fc1132f528 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:33.814198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 513/1021] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:33.831595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62716d83-1a35-450d-bcb8-8a2734b00da0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:33.832249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62716d83-1a35-450d-bcb8-8a2734b00da0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:33.832506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62716d83-1a35-450d-bcb8-8a2734b00da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:33.863287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-62716d83-1a35-450d-bcb8-8a2734b00da0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:33.865298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-62716d83-1a35-450d-bcb8-8a2734b00da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:33.997361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 509/1022] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:33 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:34.452013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a68a94-b5f0-4197-9bb0-e5fc1132f528 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:34.612732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:34.638235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a68a94-b5f0-4197-9bb0-e5fc1132f528 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:34.698554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b229e97-f618-4114-8482-1b6f47480273 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:34.699621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b229e97-f618-4114-8482-1b6f47480273 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:34.700096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b229e97-f618-4114-8482-1b6f47480273 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:34.700568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b229e97-f618-4114-8482-1b6f47480273 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:34.702521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b229e97-f618-4114-8482-1b6f47480273 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:34.705157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b229e97-f618-4114-8482-1b6f47480273 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:34.708356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 514/1023] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:34.716348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62716d83-1a35-450d-bcb8-8a2734b00da0 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:34.823842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:34.898314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62716d83-1a35-450d-bcb8-8a2734b00da0 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:34.941850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15997055-e7a7-439d-88fe-1417ffd4ebe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:34.942470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15997055-e7a7-439d-88fe-1417ffd4ebe4 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:34.942771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15997055-e7a7-439d-88fe-1417ffd4ebe4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:34.943051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15997055-e7a7-439d-88fe-1417ffd4ebe4 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:34.943731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15997055-e7a7-439d-88fe-1417ffd4ebe4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:34.945184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15997055-e7a7-439d-88fe-1417ffd4ebe4 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:34.982767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 510/1024] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:28:34 2022] POST /identity/v3/users => generated 285 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:28:34.987327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2911962-2db4-41ce-8582-25384f501fe8 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:34.987946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2911962-2db4-41ce-8582-25384f501fe8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:34.988552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2911962-2db4-41ce-8582-25384f501fe8 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:34.990660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 515/1025] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:34.999731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61ceba-44d9-43e3-9bdd-47a5ca07cefa None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:35.000020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61ceba-44d9-43e3-9bdd-47a5ca07cefa None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:35.000302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61ceba-44d9-43e3-9bdd-47a5ca07cefa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:35.031847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-8e61ceba-44d9-43e3-9bdd-47a5ca07cefa None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:35.033406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8e61ceba-44d9-43e3-9bdd-47a5ca07cefa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:35.160476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 511/1026] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:35.238100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cb9828-8d95-4c55-ab99-c857e43433ed None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:35.238936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cb9828-8d95-4c55-ab99-c857e43433ed None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:35.239647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cb9828-8d95-4c55-ab99-c857e43433ed None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:35.242704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 516/1027] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:35.250669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1153fd0-2bd0-4d0f-85cc-1eb85d2d75ae None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:35.251152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1153fd0-2bd0-4d0f-85cc-1eb85d2d75ae None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:35.251596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1153fd0-2bd0-4d0f-85cc-1eb85d2d75ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:35.282291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a1153fd0-2bd0-4d0f-85cc-1eb85d2d75ae None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:35.284200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a1153fd0-2bd0-4d0f-85cc-1eb85d2d75ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:35.360331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:35.414761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 512/1028] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:35 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:35.448118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cb9828-8d95-4c55-ab99-c857e43433ed None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:35.618858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cb9828-8d95-4c55-ab99-c857e43433ed None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:35.681212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:35.682162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:35.682648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:35.683182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] PATH_INFO: `/v3/services/image` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:35.684285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:35.687452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:35.705336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:35.707617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:35.709243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:35.714478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 517/1029] 10.222.0.77 () {56 vars in 1105 bytes} [Mon Aug 29 22:28:35 2022] GET /identity/v3/services/image => generated 86 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:35.718260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1153fd0-2bd0-4d0f-85cc-1eb85d2d75ae None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:35.745075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:35.779093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:35.887887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1153fd0-2bd0-4d0f-85cc-1eb85d2d75ae None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:35.914878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a50c9d-88ac-49e6-a809-101ce41ec8f6 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:35.938253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:35.939272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:35.939719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:35.940137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:35.940915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] RBAC: Adding query filter params (name=image) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:35.941361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:35.943060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:35.949833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 513/1030] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:35 2022] GET /identity/v3/services?name=image => generated 122 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:35.959967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:35.975944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e74f64-65f5-47cb-8750-651070bc32de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:35.977004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e74f64-65f5-47cb-8750-651070bc32de None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:35.977496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e74f64-65f5-47cb-8750-651070bc32de None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:35.977983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e74f64-65f5-47cb-8750-651070bc32de None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:35.997837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e74f64-65f5-47cb-8750-651070bc32de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-d3e74f64-65f5-47cb-8750-651070bc32de None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:36.008761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:36.009820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:36.010690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:36.011656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 518/1031] 10.222.0.77 () {56 vars in 1120 bytes} [Mon Aug 29 22:28:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:36.019702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:36.022113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e74f64-65f5-47cb-8750-651070bc32de None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:36.115472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df4e493-6d21-4256-91e0-a3fc2197c21a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:36.170810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:36.171826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:36.172117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:36.172390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:36.173034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] RBAC: Adding query filter params (type=image) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:36.173337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:36.174949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:36.181799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 514/1032] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:35 2022] GET /identity/v3/services?type=image => generated 350 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:36.190411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:36.191033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:36.224877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e74f64-65f5-47cb-8750-651070bc32de None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:36.287122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:36.288088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:36.288446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:36.288730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:36.289904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:36.290403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:36.292646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:36.309916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 519/1033] 10.222.0.77 () {56 vars in 1115 bytes} [Mon Aug 29 22:28:36 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:36.319117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:36.375456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc9bbc-cd47-4d81-a1db-2d20f22c8674 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:36.379631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:36.438079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:36.438606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:36.438927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:36.439209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:36.439938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] RBAC: Adding query filter params (interface=public, service_id=7bdd13515b8a415b9e09c6ca1072fac7, region_id=RegionOne) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:36.440253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:36.442081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:36.449997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 515/1034] 10.222.0.77 () {56 vars in 1251 bytes} [Mon Aug 29 22:28:36 2022] GET /identity/v3/endpoints?service_id=7bdd13515b8a415b9e09c6ca1072fac7&interface=public®ion_id=RegionOne => generated 194 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:36.456620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:36.518869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98abbf8-1998-4ffe-a3c7-1ab683160b41 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:36.561738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:36.562902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:36.563651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:36.564390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:36.588405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:36.588405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:36.588405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:36.599368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:36.601190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:36.602965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:36.602965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:36.602965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:36.602965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 520/1035] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:36.607710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:36.622344 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb99e0f-c7d6-4391-a240-e622cf587c62 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:36.673230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786102b4-d106-42a4-b2dc-4c6ff4c788d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:36.674029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786102b4-d106-42a4-b2dc-4c6ff4c788d8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:36.674456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786102b4-d106-42a4-b2dc-4c6ff4c788d8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:36.674959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786102b4-d106-42a4-b2dc-4c6ff4c788d8 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:36.676166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786102b4-d106-42a4-b2dc-4c6ff4c788d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:36.677739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786102b4-d106-42a4-b2dc-4c6ff4c788d8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:36.694846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 516/1036] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:28:36 2022] GET /identity/v3/services => generated 1957 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:28:36.800137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a293e-0809-4731-974a-ae055a136d08 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:36.871901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:36.872754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:36.873225 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:36.873692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:36.874761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:36.875217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:36.877190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:36.895201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 521/1037] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:36.904621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786102b4-d106-42a4-b2dc-4c6ff4c788d8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:36.962795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:37.094256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786102b4-d106-42a4-b2dc-4c6ff4c788d8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:37.156729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:37.157733 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:37.158194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:37.158634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] PATH_INFO: `/v3/users/nova` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:37.178430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:37.181599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:37.206474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:37.208278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:37.209980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 517/1038] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:28:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:37.221852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:37.335736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:37.413847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326c23a-c22a-4355-98d2-1921f69562ea None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:37.473924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7583ea-24fa-4572-9348-03323b95f883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:37.474865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7583ea-24fa-4572-9348-03323b95f883 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:37.475522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7583ea-24fa-4572-9348-03323b95f883 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:37.476091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7583ea-24fa-4572-9348-03323b95f883 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:37.477508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7583ea-24fa-4572-9348-03323b95f883 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:37.478032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7583ea-24fa-4572-9348-03323b95f883 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:37.480635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7583ea-24fa-4572-9348-03323b95f883 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:37.514108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 522/1039] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:28:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:37.524414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:37.697303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:37.716052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53308aec-439c-4af2-b8d4-20ef58fb9b26 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:37.757108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f63bbd-bd6f-4ea0-8f04-6c02cae64de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:37.757617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f63bbd-bd6f-4ea0-8f04-6c02cae64de6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:37.757903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f63bbd-bd6f-4ea0-8f04-6c02cae64de6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:37.758176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f63bbd-bd6f-4ea0-8f04-6c02cae64de6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:37.866055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f63bbd-bd6f-4ea0-8f04-6c02cae64de6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:37.868572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f63bbd-bd6f-4ea0-8f04-6c02cae64de6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:37.898782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c068c5fe-0509-4376-b2c1-2ec15c516e50 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:37.899208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c068c5fe-0509-4376-b2c1-2ec15c516e50 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:37.899363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c068c5fe-0509-4376-b2c1-2ec15c516e50 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:37.900911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 523/1040] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:37.907060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a643c25c-b019-4937-9727-edd95e21bd54 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:37.907457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a643c25c-b019-4937-9727-edd95e21bd54 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:37.907868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a643c25c-b019-4937-9727-edd95e21bd54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:37.932908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a643c25c-b019-4937-9727-edd95e21bd54 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:37.934816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a643c25c-b019-4937-9727-edd95e21bd54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:38.048777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 524/1041] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:37 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:38.057688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:38.092475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 518/1042] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:37 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 562 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:28:38.099148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a643c25c-b019-4937-9727-edd95e21bd54 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:38.189481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:38.299034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a643c25c-b019-4937-9727-edd95e21bd54 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:38.362175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:38.363548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:38.364040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:38.364529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:38.379949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:38.381976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:38.398114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:38.400033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:38.402423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:38.402423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:38.402423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:38.407354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 525/1043] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:38.409884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f63bbd-bd6f-4ea0-8f04-6c02cae64de6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:38.514337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:38.609646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f63bbd-bd6f-4ea0-8f04-6c02cae64de6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:38.655470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:38.667711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7ab1f4-1b7d-4daa-99d0-057a9010f1f8 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:38.682710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:38.683537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:38.684041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:38.684527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:38.685767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:38.686309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:38.689019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:38.710880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 519/1044] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:38.711577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005fa53f-f1ac-40a7-a15c-271a321f0e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:38.712324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005fa53f-f1ac-40a7-a15c-271a321f0e0f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:38.712611 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005fa53f-f1ac-40a7-a15c-271a321f0e0f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:38.712879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005fa53f-f1ac-40a7-a15c-271a321f0e0f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:38.717737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:38.728252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005fa53f-f1ac-40a7-a15c-271a321f0e0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-005fa53f-f1ac-40a7-a15c-271a321f0e0f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:38.733100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:38.734675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:38.740103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 526/1045] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:28:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:38.743864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005fa53f-f1ac-40a7-a15c-271a321f0e0f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:38.833542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:38.888731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005fa53f-f1ac-40a7-a15c-271a321f0e0f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:38.913717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819644fb-ee00-457f-89e9-e7a14989cfa3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:38.932231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:38.932856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:38.933137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:38.933413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:38.934458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:38.934800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:38.936524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:38.960644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 527/1046] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:28:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:38.962413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:38.969633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:38.970282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:38.970557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:38.970856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:38.978531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:38.980137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:38.992578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:38.995207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:38.996176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 520/1047] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:39.004066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:39.007004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:39.141255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0dc598-f0d8-413c-ad14-53b4340c4011 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:39.168596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa071e3-42d9-4b71-ba85-1aaf1e69ff71 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:39.185218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:39.207845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dabffc-dc8d-47d3-8547-60b5dd222583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:39.208897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dabffc-dc8d-47d3-8547-60b5dd222583 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:39.209369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dabffc-dc8d-47d3-8547-60b5dd222583 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:39.209830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dabffc-dc8d-47d3-8547-60b5dd222583 None admin] PATH_INFO: `/v3/users/65f77866cf7042948f9b64b9af6d3d61` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:39.221376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:39.222179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:39.222642 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:39.223113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:39.224016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:39.224448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:39.226206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:39.232922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dabffc-dc8d-47d3-8547-60b5dd222583 None admin] RBAC: Authorizing `identity:get_user(user_id=65f77866cf7042948f9b64b9af6d3d61)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:39.237304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dabffc-dc8d-47d3-8547-60b5dd222583 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:39.247414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 528/1048] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:28:38 2022] GET /identity/v3/users/65f77866cf7042948f9b64b9af6d3d61 => generated 285 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:39.247695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 521/1049] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:28:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:39.260273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dabffc-dc8d-47d3-8547-60b5dd222583 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:39.262523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:39.376822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:39.416650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dabffc-dc8d-47d3-8547-60b5dd222583 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:39.439042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9990be14-bba4-44e2-b328-b7d32ea2aecb None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:39.491531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d335439-c565-4228-845a-d5f1587960a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:39.492494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d335439-c565-4228-845a-d5f1587960a1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:39.492971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d335439-c565-4228-845a-d5f1587960a1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:39.493402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d335439-c565-4228-845a-d5f1587960a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:39.508810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3560c980-4474-46ff-ac33-22c8014cecee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:39.509779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3560c980-4474-46ff-ac33-22c8014cecee None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:39.510226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3560c980-4474-46ff-ac33-22c8014cecee None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:39.510663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3560c980-4474-46ff-ac33-22c8014cecee None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:39.511857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3560c980-4474-46ff-ac33-22c8014cecee None admin] RBAC: Adding query filter params (role.id=f772879e7c8a4395a3cdc078d84cc832, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=57491e4de6c3435e9d818266a910b831) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:39.512225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3560c980-4474-46ff-ac33-22c8014cecee None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:39.514668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3560c980-4474-46ff-ac33-22c8014cecee None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:39.524379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 522/1050] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:39 2022] GET /identity/v3/role_assignments?user.id=57491e4de6c3435e9d818266a910b831&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=f772879e7c8a4395a3cdc078d84cc832 => generated 259 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:39.526820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac9685f-bc23-4b90-a24f-af699e797d9f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:39.527315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac9685f-bc23-4b90-a24f-af699e797d9f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:39.527782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac9685f-bc23-4b90-a24f-af699e797d9f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:39.530384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 523/1051] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:39.539867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d83a1-dfd9-4f3c-8320-2ab95257a5e9 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:39.540298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d83a1-dfd9-4f3c-8320-2ab95257a5e9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:39.540708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d83a1-dfd9-4f3c-8320-2ab95257a5e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:39.570576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3a4d83a1-dfd9-4f3c-8320-2ab95257a5e9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:39.572484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3a4d83a1-dfd9-4f3c-8320-2ab95257a5e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:39.630070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:39.630352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d335439-c565-4228-845a-d5f1587960a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:39.632959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d335439-c565-4228-845a-d5f1587960a1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:39.707778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 524/1052] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:39 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:39.778902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:39.848010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 529/1053] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:39 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 592 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:28:39.850521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4d83a1-dfd9-4f3c-8320-2ab95257a5e9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:40.049072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4d83a1-dfd9-4f3c-8320-2ab95257a5e9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:40.116828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b2a95d-d451-46bc-86a1-76ebd7f72722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:40.117892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b2a95d-d451-46bc-86a1-76ebd7f72722 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:40.118377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b2a95d-d451-46bc-86a1-76ebd7f72722 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:40.118890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b2a95d-d451-46bc-86a1-76ebd7f72722 None admin] PATH_INFO: `/v3/projects/e640c17e71514eb5a008876cd4ac6551` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:40.121091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b2a95d-d451-46bc-86a1-76ebd7f72722 None admin] RBAC: Authorizing `identity:get_project(project_id=e640c17e71514eb5a008876cd4ac6551)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:40.123965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b2a95d-d451-46bc-86a1-76ebd7f72722 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:40.135555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 525/1054] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:28:39 2022] GET /identity/v3/projects/e640c17e71514eb5a008876cd4ac6551 => generated 311 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:40.139391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d335439-c565-4228-845a-d5f1587960a1 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:40.164184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b2a95d-d451-46bc-86a1-76ebd7f72722 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:40.235798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:40.336822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d335439-c565-4228-845a-d5f1587960a1 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:40.344284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b2a95d-d451-46bc-86a1-76ebd7f72722 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:40.389798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:40.391485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:40.391771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:40.392126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] PATH_INFO: `/v3/services/image` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:40.392740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:40.394488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:40.407294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:40.409106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 29 22:28:40.411345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:40.412924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:40.412924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:40.412924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:40.412924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:40.413845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=e640c17e71514eb5a008876cd4ac6551, user.id=65f77866cf7042948f9b64b9af6d3d61) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:40.413845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:40.413845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:40.417540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 526/1055] 10.222.0.77 () {56 vars in 1105 bytes} [Mon Aug 29 22:28:40 2022] GET /identity/v3/services/image => generated 86 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:40.419353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:40.419756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 530/1056] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:40 2022] GET /identity/v3/role_assignments?user.id=65f77866cf7042948f9b64b9af6d3d61&scope.project.id=e640c17e71514eb5a008876cd4ac6551&role.id=38171c89968943bd92d9e9758db65615 => generated 259 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:40.608755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:40.618695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30057d8f-b7e4-4183-863b-5613de4ec3e1 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:40.683646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:40.684756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:40.685205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:40.685671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:40.686888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] RBAC: Adding query filter params (name=image) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:40.688056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:40.690511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:40.700531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 527/1057] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:40 2022] GET /identity/v3/services?name=image => generated 122 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:40.709898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:40.809445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:40.934856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c2a8d-1bfa-4364-91d0-aed668d28bf4 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:40.997765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1427a8a-3706-4a92-852b-509a715614db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:40.998839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1427a8a-3706-4a92-852b-509a715614db None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:40.999351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1427a8a-3706-4a92-852b-509a715614db None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:40.999882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1427a8a-3706-4a92-852b-509a715614db None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:41.001072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1427a8a-3706-4a92-852b-509a715614db None admin] RBAC: Adding query filter params (type=image) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:41.001619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1427a8a-3706-4a92-852b-509a715614db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:41.004331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1427a8a-3706-4a92-852b-509a715614db None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:41.014483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 531/1058] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:40 2022] GET /identity/v3/services?type=image => generated 350 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:41.024083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:41.220170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd89d95-b1d8-41cf-8105-5813abbd0fe3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:41.290709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca63e9c6-a758-4d1a-968d-14faadb51325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:41.291798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca63e9c6-a758-4d1a-968d-14faadb51325 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:41.292293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca63e9c6-a758-4d1a-968d-14faadb51325 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:41.292783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca63e9c6-a758-4d1a-968d-14faadb51325 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:41.294065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca63e9c6-a758-4d1a-968d-14faadb51325 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:41.297139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca63e9c6-a758-4d1a-968d-14faadb51325 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:41.346795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 528/1059] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:28:41 2022] POST /identity/v3/endpoints => generated 327 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:28:42.311741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e697dc-3dca-4525-b1b9-d44d70d9eb10 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:42.315199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e697dc-3dca-4525-b1b9-d44d70d9eb10 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:42.315199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e697dc-3dca-4525-b1b9-d44d70d9eb10 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:42.316578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 532/1060] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:42 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:42.325460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5423a10f-dc37-4723-9720-f395c9688743 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:42.326275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5423a10f-dc37-4723-9720-f395c9688743 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:42.326508 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5423a10f-dc37-4723-9720-f395c9688743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:42.359279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5423a10f-dc37-4723-9720-f395c9688743 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:42.361387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5423a10f-dc37-4723-9720-f395c9688743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:42.384366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:42.502573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 529/1061] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:42.908104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e697dc-3dca-4525-b1b9-d44d70d9eb10 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:43.012472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5862fd-2215-4389-9ca5-e0600e229588 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:43.012768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5862fd-2215-4389-9ca5-e0600e229588 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:43.013086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5862fd-2215-4389-9ca5-e0600e229588 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:43.014755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 530/1062] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:43.021471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa08ffc-5f6d-46f7-b3b8-15834c4cbc49 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:43.021942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa08ffc-5f6d-46f7-b3b8-15834c4cbc49 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:43.022370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa08ffc-5f6d-46f7-b3b8-15834c4cbc49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:43.047314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1fa08ffc-5f6d-46f7-b3b8-15834c4cbc49 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:43.050363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1fa08ffc-5f6d-46f7-b3b8-15834c4cbc49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:43.094131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e697dc-3dca-4525-b1b9-d44d70d9eb10 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:43.096195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:43.153100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 531/1063] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:43.159944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7197c2a3-70cd-4d88-bbc9-dfa02583f1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:43.160625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197c2a3-70cd-4d88-bbc9-dfa02583f1c7 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:43.160911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197c2a3-70cd-4d88-bbc9-dfa02583f1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:43.161183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197c2a3-70cd-4d88-bbc9-dfa02583f1c7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:43.177733 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7197c2a3-70cd-4d88-bbc9-dfa02583f1c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-7197c2a3-70cd-4d88-bbc9-dfa02583f1c7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:43.183331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:43.184435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:43.186832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 533/1064] 10.222.0.77 () {56 vars in 1120 bytes} [Mon Aug 29 22:28:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:43.193898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa08ffc-5f6d-46f7-b3b8-15834c4cbc49 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:43.328463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:43.384452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa08ffc-5f6d-46f7-b3b8-15834c4cbc49 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:43.447011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:43.447996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:43.448447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:43.448845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:43.449848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:43.450293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:43.452436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:43.473225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 532/1065] 10.222.0.77 () {56 vars in 1115 bytes} [Mon Aug 29 22:28:43 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:43.475662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7197c2a3-70cd-4d88-bbc9-dfa02583f1c7 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:43.552638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:43.685224 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7197c2a3-70cd-4d88-bbc9-dfa02583f1c7 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:43.732178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52975e60-6b4c-49a0-8e5a-21a4c9efc8fd None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:43.740543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:43.741595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:43.742064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:43.742523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:43.760553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:43.762416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:43.787489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:43.789764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:43.791732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:43.791732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:43.791732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:43.791732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 534/1066] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:43.802651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:43.825616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa420ea5-7a6b-41ce-a082-912b88d891e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:43.826620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa420ea5-7a6b-41ce-a082-912b88d891e4 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:43.827108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa420ea5-7a6b-41ce-a082-912b88d891e4 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:43.827549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa420ea5-7a6b-41ce-a082-912b88d891e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:43.832527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:43.839771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:43.855924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa420ea5-7a6b-41ce-a082-912b88d891e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-aa420ea5-7a6b-41ce-a082-912b88d891e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:43.863177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:43.864468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:43.865709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 533/1067] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:28:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:43.871987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa420ea5-7a6b-41ce-a082-912b88d891e4 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:44.014415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62757b16-1f9c-4ba6-b81b-d52ec3a849f2 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:44.055210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa420ea5-7a6b-41ce-a082-912b88d891e4 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:44.074858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:44.075896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.076366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.077221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.078248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:44.078885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:44.080839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:44.099478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 535/1068] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:44.106234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:44.115769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:44.116408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.116688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.116956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.117733 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:44.118044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:44.119530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:44.131835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 534/1069] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:28:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:44.137924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:44.235459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:44.246809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:44.280522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067d0c1a-688b-421a-9906-3025b091aeb4 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:44.300425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf20aadd-c432-4ad9-9ba9-4346578cfc31 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:44.342510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a76248-91d0-40a1-b817-0334999ed70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:44.343271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a76248-91d0-40a1-b817-0334999ed70c None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.343467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a76248-91d0-40a1-b817-0334999ed70c None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.344413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a76248-91d0-40a1-b817-0334999ed70c None admin] PATH_INFO: `/v3/users/65f77866cf7042948f9b64b9af6d3d61` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.346288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af54ade-cb99-46bf-be24-02e526e266ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:44.347666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af54ade-cb99-46bf-be24-02e526e266ec None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.347942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a76248-91d0-40a1-b817-0334999ed70c None admin] RBAC: Authorizing `identity:get_user(user_id=65f77866cf7042948f9b64b9af6d3d61)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:44.348135 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af54ade-cb99-46bf-be24-02e526e266ec None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.348593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af54ade-cb99-46bf-be24-02e526e266ec None admin] PATH_INFO: `/v3/users/nova` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.349773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a76248-91d0-40a1-b817-0334999ed70c None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:44.351823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 535/1070] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:28:44 2022] GET /identity/v3/users/65f77866cf7042948f9b64b9af6d3d61 => generated 285 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:44.354361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd5c8c1-cfe8-406e-96ee-617fe69d48b9 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.354667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd5c8c1-cfe8-406e-96ee-617fe69d48b9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.355169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd5c8c1-cfe8-406e-96ee-617fe69d48b9 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.357100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 536/1071] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:44.360951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd5c8c1-cfe8-406e-96ee-617fe69d48b9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:44.363273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af54ade-cb99-46bf-be24-02e526e266ec None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:44.365158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af54ade-cb99-46bf-be24-02e526e266ec None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-4af54ade-cb99-46bf-be24-02e526e266ec None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:44.383701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:44.384843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:44.385823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 536/1072] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:28:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:44.387670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1a5364-a66f-48af-9262-cda78bd8eaf7 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.388141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1a5364-a66f-48af-9262-cda78bd8eaf7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.388556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1a5364-a66f-48af-9262-cda78bd8eaf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.417203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7c1a5364-a66f-48af-9262-cda78bd8eaf7 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:44.419099 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7c1a5364-a66f-48af-9262-cda78bd8eaf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:44.510558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd5c8c1-cfe8-406e-96ee-617fe69d48b9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:44.515452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 537/1073] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:44.516964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1a5364-a66f-48af-9262-cda78bd8eaf7 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:44.549227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:44.561323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4704c22a-5b68-4c5e-86d5-d183fbaa0540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:44.562943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4704c22a-5b68-4c5e-86d5-d183fbaa0540 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.563466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4704c22a-5b68-4c5e-86d5-d183fbaa0540 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.563966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4704c22a-5b68-4c5e-86d5-d183fbaa0540 None admin] PATH_INFO: `/v3/projects/e640c17e71514eb5a008876cd4ac6551` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.565485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4704c22a-5b68-4c5e-86d5-d183fbaa0540 None admin] RBAC: Authorizing `identity:get_project(project_id=e640c17e71514eb5a008876cd4ac6551)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:44.567286 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4704c22a-5b68-4c5e-86d5-d183fbaa0540 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:44.574278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 537/1074] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:28:44 2022] GET /identity/v3/projects/e640c17e71514eb5a008876cd4ac6551 => generated 311 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:44.576203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4704c22a-5b68-4c5e-86d5-d183fbaa0540 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:44.618107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:44.682126 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1a5364-a66f-48af-9262-cda78bd8eaf7 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:44.734424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:44.735274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.735703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.736073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.736843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:44.737176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:44.738993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:44.757456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4704c22a-5b68-4c5e-86d5-d183fbaa0540 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:44.762773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 538/1075] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:28:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:44.772420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:44.823736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4300e9d8-6b9c-43b8-870f-ebf4bee57f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:44.824935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300e9d8-6b9c-43b8-870f-ebf4bee57f77 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:44.825389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300e9d8-6b9c-43b8-870f-ebf4bee57f77 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:44.825831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300e9d8-6b9c-43b8-870f-ebf4bee57f77 None admin] PATH_INFO: `/v3/projects/e640c17e71514eb5a008876cd4ac6551/users/65f77866cf7042948f9b64b9af6d3d61/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:44.831408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4300e9d8-6b9c-43b8-870f-ebf4bee57f77 None admin] RBAC: Authorizing `identity:create_grant(project_id=e640c17e71514eb5a008876cd4ac6551, user_id=65f77866cf7042948f9b64b9af6d3d61, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:44.834245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4300e9d8-6b9c-43b8-870f-ebf4bee57f77 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:44.855063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 538/1076] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:28:44 2022] PUT /identity/v3/projects/e640c17e71514eb5a008876cd4ac6551/users/65f77866cf7042948f9b64b9af6d3d61/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:28:44.856178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4300e9d8-6b9c-43b8-870f-ebf4bee57f77 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:44.971878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339feb3-d3fb-44a2-977b-ce0838d04006 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:44.976727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:45.020750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:45.048248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:45.048969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:45.049239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:45.049510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:45.063210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:45.066200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:45.071147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4300e9d8-6b9c-43b8-870f-ebf4bee57f77 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:45.092238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:45.093858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:45.095473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:45.095473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:45.095473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:45.095473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 539/1077] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:45.103068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:45.146636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c4a493-65a7-4189-b454-7f3479706919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:45.147840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c4a493-65a7-4189-b454-7f3479706919 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:45.148289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c4a493-65a7-4189-b454-7f3479706919 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:45.148627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c4a493-65a7-4189-b454-7f3479706919 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:45.267423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5891a0-5bf0-4e85-a688-578a155dd062 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:45.280461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c4a493-65a7-4189-b454-7f3479706919 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:45.283223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c4a493-65a7-4189-b454-7f3479706919 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:45.328140 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:45.329138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:45.329606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:45.330178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:45.331543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:45.332167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:45.334603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:45.356253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 540/1078] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:45.376657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:45.397885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:45.441070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:45.501567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 539/1079] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 646 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:28:45.511050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c4a493-65a7-4189-b454-7f3479706919 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:45.580872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356ddb64-52ca-4b77-9c5b-1ea99ec6fb76 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:45.643064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:45.644525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:45.644909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:45.645349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:45.658129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:45.660551 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:45.683595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:45.685404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:45.688499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 541/1080] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:45.695154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:45.712960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c4a493-65a7-4189-b454-7f3479706919 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:45.782052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:45.782994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:45.783399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:45.783805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:45.801284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:45.803169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:45.836357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:45.838225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:45.840696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:45.840696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:45.840696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:45.845815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 540/1081] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:45.847158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:45.861811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:45.871123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:45.881542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4688a31d-9a1f-438f-9b6f-7a68323ebc9c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:45.925830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:45.926376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:45.926642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:45.926937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:45.927640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:45.927947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:45.929383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:45.945883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 542/1082] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:28:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:45.960124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:46.010142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a81ae-1d2f-411f-a043-6585ba470abb None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:46.073526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2bb5d8-f5c6-4bab-89a4-1492f94ce0ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:46.074705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2bb5d8-f5c6-4bab-89a4-1492f94ce0ce None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:46.075340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2bb5d8-f5c6-4bab-89a4-1492f94ce0ce None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:46.075933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2bb5d8-f5c6-4bab-89a4-1492f94ce0ce None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:46.077137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2bb5d8-f5c6-4bab-89a4-1492f94ce0ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:46.077732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2bb5d8-f5c6-4bab-89a4-1492f94ce0ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:46.080927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2bb5d8-f5c6-4bab-89a4-1492f94ce0ce None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:46.114786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 541/1083] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:46.123654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b9b829-8811-4ef9-a995-e3a08cd86a4d None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:46.189902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8aacd15-f27a-4025-bfdc-c1db8de9ef43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:46.190901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8aacd15-f27a-4025-bfdc-c1db8de9ef43 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:46.191384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8aacd15-f27a-4025-bfdc-c1db8de9ef43 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:46.191866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8aacd15-f27a-4025-bfdc-c1db8de9ef43 None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/57491e4de6c3435e9d818266a910b831/roles/f772879e7c8a4395a3cdc078d84cc832` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:46.195439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8aacd15-f27a-4025-bfdc-c1db8de9ef43 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=57491e4de6c3435e9d818266a910b831, role_id=f772879e7c8a4395a3cdc078d84cc832)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:46.198111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8aacd15-f27a-4025-bfdc-c1db8de9ef43 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:46.221171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 543/1084] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:28:45 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/57491e4de6c3435e9d818266a910b831/roles/f772879e7c8a4395a3cdc078d84cc832 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:28:47.032169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07a434-8698-428a-91b2-2ff49eaed184 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:47.032764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07a434-8698-428a-91b2-2ff49eaed184 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:47.032764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07a434-8698-428a-91b2-2ff49eaed184 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:47.034276 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 542/1085] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:47.041898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f694e5b-26c6-4c22-9626-98fa4a0ca86d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:47.042548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f694e5b-26c6-4c22-9626-98fa4a0ca86d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:47.042906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f694e5b-26c6-4c22-9626-98fa4a0ca86d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:47.074319 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1f694e5b-26c6-4c22-9626-98fa4a0ca86d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:47.075654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1f694e5b-26c6-4c22-9626-98fa4a0ca86d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:47.088438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:47.191501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 544/1086] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:47.522820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea07a434-8698-428a-91b2-2ff49eaed184 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:47.627024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:47.673879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea07a434-8698-428a-91b2-2ff49eaed184 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:47.716969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68481a8b-a17b-40bf-ac32-590917618801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:47.717649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68481a8b-a17b-40bf-ac32-590917618801 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:47.717944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68481a8b-a17b-40bf-ac32-590917618801 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:47.718201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68481a8b-a17b-40bf-ac32-590917618801 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:47.731384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68481a8b-a17b-40bf-ac32-590917618801 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-68481a8b-a17b-40bf-ac32-590917618801 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:47.737153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:47.743204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:47.745223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 543/1087] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:28:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:47.750240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f694e5b-26c6-4c22-9626-98fa4a0ca86d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:47.892072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f694e5b-26c6-4c22-9626-98fa4a0ca86d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:47.952062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:47.953136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:47.954156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:47.954629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:47.956232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:47.956785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:47.959596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:47.979116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 545/1088] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:28:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:47.988905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68481a8b-a17b-40bf-ac32-590917618801 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:48.144555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68481a8b-a17b-40bf-ac32-590917618801 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:48.187160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:48.201958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ebde7f-8975-485b-8254-04c5de6f472e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:48.202932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ebde7f-8975-485b-8254-04c5de6f472e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:48.203931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ebde7f-8975-485b-8254-04c5de6f472e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:48.204413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ebde7f-8975-485b-8254-04c5de6f472e None admin] PATH_INFO: `/v3/users/65f77866cf7042948f9b64b9af6d3d61` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:48.206600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ebde7f-8975-485b-8254-04c5de6f472e None admin] RBAC: Authorizing `identity:get_user(user_id=65f77866cf7042948f9b64b9af6d3d61)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:48.208810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ebde7f-8975-485b-8254-04c5de6f472e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:48.211676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 544/1089] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:28:47 2022] GET /identity/v3/users/65f77866cf7042948f9b64b9af6d3d61 => generated 285 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:48.221714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:48.237519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:48.390192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ae42bf-335f-43bf-833a-933c8c762f00 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:48.440043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd52f95-3a3f-4224-880f-a4d90c9ff2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:48.440663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd52f95-3a3f-4224-880f-a4d90c9ff2c4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:48.440940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd52f95-3a3f-4224-880f-a4d90c9ff2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:48.441216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd52f95-3a3f-4224-880f-a4d90c9ff2c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:48.498184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f2a9d4-c562-4b22-aa15-a11c671a75af None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:48.498595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f2a9d4-c562-4b22-aa15-a11c671a75af None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:48.499066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f2a9d4-c562-4b22-aa15-a11c671a75af None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:48.502100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 545/1090] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:48.510203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bf1d4-4b08-462f-9cad-faccf0bb0d0f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:48.510636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bf1d4-4b08-462f-9cad-faccf0bb0d0f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:48.511102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bf1d4-4b08-462f-9cad-faccf0bb0d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:48.537616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd52f95-3a3f-4224-880f-a4d90c9ff2c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:48.539881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1c6bf1d4-4b08-462f-9cad-faccf0bb0d0f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:48.540060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd52f95-3a3f-4224-880f-a4d90c9ff2c4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:48.541979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1c6bf1d4-4b08-462f-9cad-faccf0bb0d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:48.549380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:48.633862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 546/1091] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:48.677648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 546/1092] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:48 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:28:48.684388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bf1d4-4b08-462f-9cad-faccf0bb0d0f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:48.850617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bf1d4-4b08-462f-9cad-faccf0bb0d0f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:48.891824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:48.904866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f06b8a9-d203-4933-b687-1e7cf84d6706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:48.905609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f06b8a9-d203-4933-b687-1e7cf84d6706 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:48.905985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f06b8a9-d203-4933-b687-1e7cf84d6706 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:48.906375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f06b8a9-d203-4933-b687-1e7cf84d6706 None admin] PATH_INFO: `/v3/projects/e640c17e71514eb5a008876cd4ac6551` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:48.907710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f06b8a9-d203-4933-b687-1e7cf84d6706 None admin] RBAC: Authorizing `identity:get_project(project_id=e640c17e71514eb5a008876cd4ac6551)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:48.909263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f06b8a9-d203-4933-b687-1e7cf84d6706 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:48.914971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 547/1093] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:28:48 2022] GET /identity/v3/projects/e640c17e71514eb5a008876cd4ac6551 => generated 311 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:48.917596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd52f95-3a3f-4224-880f-a4d90c9ff2c4 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:48.984171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f06b8a9-d203-4933-b687-1e7cf84d6706 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:49.067530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd52f95-3a3f-4224-880f-a4d90c9ff2c4 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:49.077686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:49.110127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:49.110840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:49.111220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:49.111975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:49.112661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=e640c17e71514eb5a008876cd4ac6551, user.id=65f77866cf7042948f9b64b9af6d3d61) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:49.113114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:49.114714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:49.122100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 547/1094] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:48 2022] GET /identity/v3/role_assignments?user.id=65f77866cf7042948f9b64b9af6d3d61&scope.project.id=e640c17e71514eb5a008876cd4ac6551&role.id=38171c89968943bd92d9e9758db65615 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:49.155723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f06b8a9-d203-4933-b687-1e7cf84d6706 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:49.224131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8beba950-5e8f-42ea-b2a3-bb30a40360c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:49.225247 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beba950-5e8f-42ea-b2a3-bb30a40360c9 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:49.225661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beba950-5e8f-42ea-b2a3-bb30a40360c9 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:49.225984 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beba950-5e8f-42ea-b2a3-bb30a40360c9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:49.242698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8beba950-5e8f-42ea-b2a3-bb30a40360c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-8beba950-5e8f-42ea-b2a3-bb30a40360c9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:28:49.247667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:49.249464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:49.251844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 548/1095] 10.222.0.77 () {56 vars in 1120 bytes} [Mon Aug 29 22:28:48 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:28:49.256988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:49.406105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34051a4-cac1-48a3-b201-a684544c2220 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:49.452152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:49.452775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:49.453053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:49.453936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:49.454792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:49.455123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:49.456427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:49.469677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 548/1096] 10.222.0.77 () {56 vars in 1115 bytes} [Mon Aug 29 22:28:49 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:49.475948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8beba950-5e8f-42ea-b2a3-bb30a40360c9 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:49.552183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:49.630337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8beba950-5e8f-42ea-b2a3-bb30a40360c9 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:49.686822 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:49.687357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:49.687663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:49.687945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:49.700261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:49.702845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:49.727548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:49.729079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:49.736694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:49.736694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:49.736694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:49.736694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 549/1097] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:49.736998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:49.784695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:49.886335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abfa52c-ea6a-4627-a515-bba399f9f26f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:49.951469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:49.952388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:49.952867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:49.953336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:49.954507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:49.955148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:49.957537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:49.972790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 549/1098] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:49.978833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:50.091285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0da4a5-47dc-49c6-892d-b3ba7a542410 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:50.093988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:50.133377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:50.134421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:50.134877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:50.135364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] PATH_INFO: `/v3/users/nova` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:50.148225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:50.150615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:28:50.169595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:50.171312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 29 22:28:50.172819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:50.174954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 550/1099] 10.222.0.77 () {56 vars in 1093 bytes} [Mon Aug 29 22:28:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:50.180766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:50.306225 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:50.322230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa2882d-6d97-491f-b428-62be3e3626ea None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:50.365060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:50.365662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:50.365941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:50.366211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:50.369521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:50.371310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:50.379223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:50.415620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 550/1100] 10.222.0.77 () {56 vars in 1133 bytes} [Mon Aug 29 22:28:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:50.421830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:50.584550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67921345-91a5-40ef-be10-e990566ff9ff None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:50.636097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869b4869-4579-49fd-8ba7-faee53fa53ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:50.637217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869b4869-4579-49fd-8ba7-faee53fa53ce None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:50.637702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869b4869-4579-49fd-8ba7-faee53fa53ce None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:50.638143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869b4869-4579-49fd-8ba7-faee53fa53ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:50.681860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:50.740235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869b4869-4579-49fd-8ba7-faee53fa53ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:50.742911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869b4869-4579-49fd-8ba7-faee53fa53ce None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:50.878816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 551/1101] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:28:50.884814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:51.024057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0611637e-a75e-4263-a7e6-8e5aa3b62d24 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:51.080416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e97fad-db8e-4163-ace8-764d66de05d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:51.081052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e97fad-db8e-4163-ace8-764d66de05d4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:51.081329 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e97fad-db8e-4163-ace8-764d66de05d4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:51.081594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e97fad-db8e-4163-ace8-764d66de05d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:51.095451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e97fad-db8e-4163-ace8-764d66de05d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:51.097572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e97fad-db8e-4163-ace8-764d66de05d4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-f1e97fad-db8e-4163-ace8-764d66de05d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:28:51.118438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:51.119941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:28:51.122208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:28:51.122208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:28:51.122208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:28:51.122208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 551/1102] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:51.128635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869b4869-4579-49fd-8ba7-faee53fa53ce None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:51.214205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:51.241915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f26d7-b52e-47ce-a25d-82f8791855b9 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:51.242433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f26d7-b52e-47ce-a25d-82f8791855b9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:51.242834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f26d7-b52e-47ce-a25d-82f8791855b9 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:51.244444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 552/1103] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:51.250799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed16c8d-3544-46ef-8a17-5649d67a1c16 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:51.251224 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed16c8d-3544-46ef-8a17-5649d67a1c16 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:51.251635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed16c8d-3544-46ef-8a17-5649d67a1c16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:51.258229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:51.277406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-4ed16c8d-3544-46ef-8a17-5649d67a1c16 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:51.279261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4ed16c8d-3544-46ef-8a17-5649d67a1c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:51.282435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869b4869-4579-49fd-8ba7-faee53fa53ce None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:51.344678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:51.345457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:51.346035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:51.346421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:51.347225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:51.347626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:51.349447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:51.368722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 552/1104] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:28:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:51.370319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:51.386182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 553/1105] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:28:51.509836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef68325-abef-4f7f-9cd5-719cc67acc11 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:51.552362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:51.566300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:51.566964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:51.567248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:51.567520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:51.576585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:51.579844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:28:51.601081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:28:51.602753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:28:51.604789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 553/1106] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:28:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:28:51.612331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed16c8d-3544-46ef-8a17-5649d67a1c16 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:51.707187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:51.725732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed16c8d-3544-46ef-8a17-5649d67a1c16 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:51.771154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:51.777052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:51.777693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:51.777971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:51.778242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:51.778996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:51.779307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:51.780767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:51.800566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 554/1107] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:28:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:51.809517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:51.865104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e833129-25df-4dd1-8e15-1c9c053cbcfc None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:51.919919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aafab17-85d7-477e-8706-332f4e5e3e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:51.920558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aafab17-85d7-477e-8706-332f4e5e3e00 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:51.920840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aafab17-85d7-477e-8706-332f4e5e3e00 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:51.921110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aafab17-85d7-477e-8706-332f4e5e3e00 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:51.922296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aafab17-85d7-477e-8706-332f4e5e3e00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:51.923778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aafab17-85d7-477e-8706-332f4e5e3e00 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:51.925393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 554/1108] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:28:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:51.931709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aafab17-85d7-477e-8706-332f4e5e3e00 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:51.957419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9389d5e0-c2cb-4e6f-86c0-120f4c739efb None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:52.000439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd35989-4edf-4ab1-b564-acd65be51383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:52.001533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd35989-4edf-4ab1-b564-acd65be51383 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:52.001920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd35989-4edf-4ab1-b564-acd65be51383 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:52.002307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd35989-4edf-4ab1-b564-acd65be51383 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:52.003259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd35989-4edf-4ab1-b564-acd65be51383 None admin] RBAC: Adding query filter params (role.id=f772879e7c8a4395a3cdc078d84cc832, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=57491e4de6c3435e9d818266a910b831) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:52.003573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd35989-4edf-4ab1-b564-acd65be51383 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:52.004707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:52.005195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd35989-4edf-4ab1-b564-acd65be51383 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:52.013334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 555/1109] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:51 2022] GET /identity/v3/role_assignments?user.id=57491e4de6c3435e9d818266a910b831&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=f772879e7c8a4395a3cdc078d84cc832 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:52.077274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aafab17-85d7-477e-8706-332f4e5e3e00 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:52.135554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcadea82-10b6-4122-9d4b-128aa474b928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:52.136511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcadea82-10b6-4122-9d4b-128aa474b928 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:52.136948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcadea82-10b6-4122-9d4b-128aa474b928 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:52.137372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcadea82-10b6-4122-9d4b-128aa474b928 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:52.138481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcadea82-10b6-4122-9d4b-128aa474b928 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:52.141588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcadea82-10b6-4122-9d4b-128aa474b928 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:52.174838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 555/1110] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:28:51 2022] POST /identity/v3/users => generated 286 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:28:54.430435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d1d79-65dc-4ec5-8554-e2460ccb9404 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:54.431090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d1d79-65dc-4ec5-8554-e2460ccb9404 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:54.431090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d1d79-65dc-4ec5-8554-e2460ccb9404 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:54.432536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 556/1111] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:28:54.438209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58edbbc-c450-4dd5-ad4a-084b5ccd3a41 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:54.438904 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58edbbc-c450-4dd5-ad4a-084b5ccd3a41 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:54.438904 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58edbbc-c450-4dd5-ad4a-084b5ccd3a41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:54.461825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f58edbbc-c450-4dd5-ad4a-084b5ccd3a41 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:54.463129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f58edbbc-c450-4dd5-ad4a-084b5ccd3a41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:54.547367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 556/1112] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:54.829241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793d1d79-65dc-4ec5-8554-e2460ccb9404 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:54.895436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:54.945783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793d1d79-65dc-4ec5-8554-e2460ccb9404 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:54.999489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93380145-487b-4ace-98aa-1ddddd764553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:55.000141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93380145-487b-4ace-98aa-1ddddd764553 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:55.000447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93380145-487b-4ace-98aa-1ddddd764553 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:55.000728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93380145-487b-4ace-98aa-1ddddd764553 None admin] PATH_INFO: `/v3/roles/f6e14365f5f54e2da132ac99f92451f5` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:55.002180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93380145-487b-4ace-98aa-1ddddd764553 None admin] RBAC: Authorizing `identity:get_role(role_id=f6e14365f5f54e2da132ac99f92451f5)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:55.003662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93380145-487b-4ace-98aa-1ddddd764553 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:55.005803 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 557/1113] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:28:54 2022] GET /identity/v3/roles/f6e14365f5f54e2da132ac99f92451f5 => generated 223 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:28:55.014294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58edbbc-c450-4dd5-ad4a-084b5ccd3a41 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:55.061809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:55.127974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58edbbc-c450-4dd5-ad4a-084b5ccd3a41 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:55.164628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03763a9e-d5c2-44fc-a0ed-310fe876ec35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:55.165260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03763a9e-d5c2-44fc-a0ed-310fe876ec35 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:55.165543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03763a9e-d5c2-44fc-a0ed-310fe876ec35 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:55.165819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03763a9e-d5c2-44fc-a0ed-310fe876ec35 None admin] PATH_INFO: `/v3/users/b34f7f2a91d34384aa2bc9ecffa495b7` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:55.180520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03763a9e-d5c2-44fc-a0ed-310fe876ec35 None admin] RBAC: Authorizing `identity:get_user(user_id=b34f7f2a91d34384aa2bc9ecffa495b7)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:55.182542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03763a9e-d5c2-44fc-a0ed-310fe876ec35 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:55.184559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 557/1114] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:28:55 2022] GET /identity/v3/users/b34f7f2a91d34384aa2bc9ecffa495b7 => generated 286 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:55.191362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93380145-487b-4ace-98aa-1ddddd764553 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:55.321187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93380145-487b-4ace-98aa-1ddddd764553 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:55.363622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:55.381902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e42358-29a7-4ea2-9620-64fcf5a9534a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:55.382596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e42358-29a7-4ea2-9620-64fcf5a9534a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:55.382997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e42358-29a7-4ea2-9620-64fcf5a9534a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:55.383352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e42358-29a7-4ea2-9620-64fcf5a9534a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:55.477386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e42358-29a7-4ea2-9620-64fcf5a9534a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:55.479518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e42358-29a7-4ea2-9620-64fcf5a9534a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:55.604358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 558/1115] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:28:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 22:28:55.611280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03763a9e-d5c2-44fc-a0ed-310fe876ec35 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:55.712494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:55.718708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03763a9e-d5c2-44fc-a0ed-310fe876ec35 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:55.757131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235dbb5f-cb50-4261-8e2c-a156d6a67473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:55.757776 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235dbb5f-cb50-4261-8e2c-a156d6a67473 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:55.758058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235dbb5f-cb50-4261-8e2c-a156d6a67473 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:55.758330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235dbb5f-cb50-4261-8e2c-a156d6a67473 None admin] PATH_INFO: `/v3/projects/e640c17e71514eb5a008876cd4ac6551` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:55.759691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235dbb5f-cb50-4261-8e2c-a156d6a67473 None admin] RBAC: Authorizing `identity:get_project(project_id=e640c17e71514eb5a008876cd4ac6551)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:55.762156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235dbb5f-cb50-4261-8e2c-a156d6a67473 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:55.763854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 558/1116] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:28:55 2022] GET /identity/v3/projects/e640c17e71514eb5a008876cd4ac6551 => generated 311 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:55.771202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e42358-29a7-4ea2-9620-64fcf5a9534a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:55.812147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:55.916693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e42358-29a7-4ea2-9620-64fcf5a9534a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:55.956322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6691733-6981-4524-a0ea-5caeae3ae948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:55.956922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6691733-6981-4524-a0ea-5caeae3ae948 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:55.957203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6691733-6981-4524-a0ea-5caeae3ae948 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:55.957472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6691733-6981-4524-a0ea-5caeae3ae948 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:55.958207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6691733-6981-4524-a0ea-5caeae3ae948 None admin] RBAC: Adding query filter params (role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=e640c17e71514eb5a008876cd4ac6551, user.id=b34f7f2a91d34384aa2bc9ecffa495b7) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:28:55.958518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6691733-6981-4524-a0ea-5caeae3ae948 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:55.959986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6691733-6981-4524-a0ea-5caeae3ae948 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:55.966197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 559/1117] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:28:55 2022] GET /identity/v3/role_assignments?user.id=b34f7f2a91d34384aa2bc9ecffa495b7&scope.project.id=e640c17e71514eb5a008876cd4ac6551&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:57.933086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb8b80-981a-4691-aa63-c84f263ce4d7 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:57.933717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb8b80-981a-4691-aa63-c84f263ce4d7 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:57.933717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb8b80-981a-4691-aa63-c84f263ce4d7 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:57.935153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 559/1118] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:28:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:28:57.942502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571c92a-5c50-4e1a-9eb1-80106a870f52 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:57.943295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571c92a-5c50-4e1a-9eb1-80106a870f52 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:57.943490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4571c92a-5c50-4e1a-9eb1-80106a870f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:57.970183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-4571c92a-5c50-4e1a-9eb1-80106a870f52 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:28:57.971615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4571c92a-5c50-4e1a-9eb1-80106a870f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:28:58.061242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 560/1119] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:28:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:28:58.350834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbb8b80-981a-4691-aa63-c84f263ce4d7 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:58.480242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbb8b80-981a-4691-aa63-c84f263ce4d7 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:58.525054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922660dc-8f8b-4f79-8464-1974134e833a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:58.525712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922660dc-8f8b-4f79-8464-1974134e833a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:58.526020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922660dc-8f8b-4f79-8464-1974134e833a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:58.526291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922660dc-8f8b-4f79-8464-1974134e833a None admin] PATH_INFO: `/v3/roles/f6e14365f5f54e2da132ac99f92451f5` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:58.527931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922660dc-8f8b-4f79-8464-1974134e833a None admin] RBAC: Authorizing `identity:get_role(role_id=f6e14365f5f54e2da132ac99f92451f5)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:58.529449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922660dc-8f8b-4f79-8464-1974134e833a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:58.530839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 560/1120] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:28:58 2022] GET /identity/v3/roles/f6e14365f5f54e2da132ac99f92451f5 => generated 223 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:58.537367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4571c92a-5c50-4e1a-9eb1-80106a870f52 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:58.564031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:58.683516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4571c92a-5c50-4e1a-9eb1-80106a870f52 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:58.734493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0644191-5c76-4fab-b681-7744a81ab85a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:58.735714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0644191-5c76-4fab-b681-7744a81ab85a None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:58.735997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0644191-5c76-4fab-b681-7744a81ab85a None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:58.736272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0644191-5c76-4fab-b681-7744a81ab85a None admin] PATH_INFO: `/v3/users/b34f7f2a91d34384aa2bc9ecffa495b7` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:58.737517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0644191-5c76-4fab-b681-7744a81ab85a None admin] RBAC: Authorizing `identity:get_user(user_id=b34f7f2a91d34384aa2bc9ecffa495b7)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:58.739057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0644191-5c76-4fab-b681-7744a81ab85a None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:58.740777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 561/1121] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:28:58 2022] GET /identity/v3/users/b34f7f2a91d34384aa2bc9ecffa495b7 => generated 286 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:58.749838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922660dc-8f8b-4f79-8464-1974134e833a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:58.785649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:58.881955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922660dc-8f8b-4f79-8464-1974134e833a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:58.918829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297a97c3-c273-4bad-a16a-3a17ee24522d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:58.919465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297a97c3-c273-4bad-a16a-3a17ee24522d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:58.919735 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297a97c3-c273-4bad-a16a-3a17ee24522d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:58.920008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297a97c3-c273-4bad-a16a-3a17ee24522d None admin] PATH_INFO: `/v3/projects/e640c17e71514eb5a008876cd4ac6551` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:58.921154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297a97c3-c273-4bad-a16a-3a17ee24522d None admin] RBAC: Authorizing `identity:get_project(project_id=e640c17e71514eb5a008876cd4ac6551)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:58.923185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297a97c3-c273-4bad-a16a-3a17ee24522d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:58.924843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 561/1122] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:28:58 2022] GET /identity/v3/projects/e640c17e71514eb5a008876cd4ac6551 => generated 311 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:28:58.931679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0644191-5c76-4fab-b681-7744a81ab85a None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:28:59.032539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:28:59.063552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0644191-5c76-4fab-b681-7744a81ab85a None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:28:59.101265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b39733-87af-42d3-aab2-21ef5332278f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:28:59.101987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b39733-87af-42d3-aab2-21ef5332278f None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:28:59.102267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b39733-87af-42d3-aab2-21ef5332278f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:28:59.102541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b39733-87af-42d3-aab2-21ef5332278f None admin] PATH_INFO: `/v3/projects/e640c17e71514eb5a008876cd4ac6551/users/b34f7f2a91d34384aa2bc9ecffa495b7/roles/f6e14365f5f54e2da132ac99f92451f5` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:28:59.104816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b39733-87af-42d3-aab2-21ef5332278f None admin] RBAC: Authorizing `identity:create_grant(project_id=e640c17e71514eb5a008876cd4ac6551, user_id=b34f7f2a91d34384aa2bc9ecffa495b7, role_id=f6e14365f5f54e2da132ac99f92451f5)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:28:59.106267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b39733-87af-42d3-aab2-21ef5332278f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:28:59.123403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 562/1123] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:28:58 2022] PUT /identity/v3/projects/e640c17e71514eb5a008876cd4ac6551/users/b34f7f2a91d34384aa2bc9ecffa495b7/roles/f6e14365f5f54e2da132ac99f92451f5 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:29:01.072862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9436072-9d56-4f9d-b66b-1a5c56aa18ef None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:01.073394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9436072-9d56-4f9d-b66b-1a5c56aa18ef None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:01.073394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9436072-9d56-4f9d-b66b-1a5c56aa18ef None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:01.074633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 562/1124] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:01.080691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce319c7-342b-4828-b44a-463776505026 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:01.081338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce319c7-342b-4828-b44a-463776505026 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:01.081338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce319c7-342b-4828-b44a-463776505026 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:01.103458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-fce319c7-342b-4828-b44a-463776505026 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:01.104689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-fce319c7-342b-4828-b44a-463776505026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:01.209911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 563/1125] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:01.488764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9436072-9d56-4f9d-b66b-1a5c56aa18ef None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:01.628025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:01.632641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9436072-9d56-4f9d-b66b-1a5c56aa18ef None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:01.695176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79881ac9-48e5-4ffc-8e84-7592831c820a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:01.695843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79881ac9-48e5-4ffc-8e84-7592831c820a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:01.696122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79881ac9-48e5-4ffc-8e84-7592831c820a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:01.696393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79881ac9-48e5-4ffc-8e84-7592831c820a None admin] PATH_INFO: `/v3/roles/f6e14365f5f54e2da132ac99f92451f5` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:01.697935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79881ac9-48e5-4ffc-8e84-7592831c820a None admin] RBAC: Authorizing `identity:get_role(role_id=f6e14365f5f54e2da132ac99f92451f5)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:01.699472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79881ac9-48e5-4ffc-8e84-7592831c820a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:01.700880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 563/1126] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:29:01 2022] GET /identity/v3/roles/f6e14365f5f54e2da132ac99f92451f5 => generated 223 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:01.710204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce319c7-342b-4828-b44a-463776505026 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:01.803031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:01.835656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce319c7-342b-4828-b44a-463776505026 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:01.874297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57efaf14-638e-4e0b-9a73-93534b88fe34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:01.874977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57efaf14-638e-4e0b-9a73-93534b88fe34 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:01.875253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57efaf14-638e-4e0b-9a73-93534b88fe34 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:01.875522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57efaf14-638e-4e0b-9a73-93534b88fe34 None admin] PATH_INFO: `/v3/users/b34f7f2a91d34384aa2bc9ecffa495b7` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:01.877324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57efaf14-638e-4e0b-9a73-93534b88fe34 None admin] RBAC: Authorizing `identity:get_user(user_id=b34f7f2a91d34384aa2bc9ecffa495b7)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:01.878833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57efaf14-638e-4e0b-9a73-93534b88fe34 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:01.880526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 564/1127] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:29:01 2022] GET /identity/v3/users/b34f7f2a91d34384aa2bc9ecffa495b7 => generated 286 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:01.886950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79881ac9-48e5-4ffc-8e84-7592831c820a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:02.026456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79881ac9-48e5-4ffc-8e84-7592831c820a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:02.065070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361b79be-306b-442b-b503-8a707aa63c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:02.065601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361b79be-306b-442b-b503-8a707aa63c0a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:02.065877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361b79be-306b-442b-b503-8a707aa63c0a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:02.066151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361b79be-306b-442b-b503-8a707aa63c0a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:02.111497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:02.154816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361b79be-306b-442b-b503-8a707aa63c0a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:02.156300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361b79be-306b-442b-b503-8a707aa63c0a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:02.288434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 564/1128] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:29:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:29:02.294989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57efaf14-638e-4e0b-9a73-93534b88fe34 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:02.409781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57efaf14-638e-4e0b-9a73-93534b88fe34 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:02.446402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5600cb2-6d94-4c20-ad16-c69d2fd3ccf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:02.447063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5600cb2-6d94-4c20-ad16-c69d2fd3ccf0 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:02.447344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5600cb2-6d94-4c20-ad16-c69d2fd3ccf0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:02.447617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5600cb2-6d94-4c20-ad16-c69d2fd3ccf0 None admin] PATH_INFO: `/v3/projects/e640c17e71514eb5a008876cd4ac6551` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:02.448766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5600cb2-6d94-4c20-ad16-c69d2fd3ccf0 None admin] RBAC: Authorizing `identity:get_project(project_id=e640c17e71514eb5a008876cd4ac6551)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:02.450214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5600cb2-6d94-4c20-ad16-c69d2fd3ccf0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:02.451928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 565/1129] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:29:02 2022] GET /identity/v3/projects/e640c17e71514eb5a008876cd4ac6551 => generated 311 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:02.460348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361b79be-306b-442b-b503-8a707aa63c0a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:02.538082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:02.581468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361b79be-306b-442b-b503-8a707aa63c0a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:02.632619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bea0411-b38c-4576-a31f-92d3c3ae08f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:02.633215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea0411-b38c-4576-a31f-92d3c3ae08f5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:02.633490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea0411-b38c-4576-a31f-92d3c3ae08f5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:02.633758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea0411-b38c-4576-a31f-92d3c3ae08f5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:02.634485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea0411-b38c-4576-a31f-92d3c3ae08f5 None admin] RBAC: Adding query filter params (role.id=f6e14365f5f54e2da132ac99f92451f5, scope.project.id=e640c17e71514eb5a008876cd4ac6551, user.id=b34f7f2a91d34384aa2bc9ecffa495b7) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:02.634819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea0411-b38c-4576-a31f-92d3c3ae08f5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:02.636272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea0411-b38c-4576-a31f-92d3c3ae08f5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:02.643286 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 565/1130] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:29:02 2022] GET /identity/v3/role_assignments?user.id=b34f7f2a91d34384aa2bc9ecffa495b7&scope.project.id=e640c17e71514eb5a008876cd4ac6551&role.id=f6e14365f5f54e2da132ac99f92451f5 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:04.699578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85b98a-9fa9-4d85-8e35-6a99ce2c46d6 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:04.700507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85b98a-9fa9-4d85-8e35-6a99ce2c46d6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:04.700507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85b98a-9fa9-4d85-8e35-6a99ce2c46d6 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:04.702152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 566/1131] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:04.709094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd16f2b-136a-42b7-a566-d57d9eacc4a2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:04.709604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd16f2b-136a-42b7-a566-d57d9eacc4a2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:04.709721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd16f2b-136a-42b7-a566-d57d9eacc4a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:04.729010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3bd16f2b-136a-42b7-a566-d57d9eacc4a2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:04.730234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3bd16f2b-136a-42b7-a566-d57d9eacc4a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:04.814120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 566/1132] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:05.101243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df85b98a-9fa9-4d85-8e35-6a99ce2c46d6 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:05.123095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:05.229907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df85b98a-9fa9-4d85-8e35-6a99ce2c46d6 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:05.270343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf123db-6109-4609-850c-354020b45a39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:05.271276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf123db-6109-4609-850c-354020b45a39 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:05.271641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf123db-6109-4609-850c-354020b45a39 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:05.271963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf123db-6109-4609-850c-354020b45a39 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:05.273244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf123db-6109-4609-850c-354020b45a39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:05.274796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf123db-6109-4609-850c-354020b45a39 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:05.276694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 567/1133] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:29:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:05.283962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd16f2b-136a-42b7-a566-d57d9eacc4a2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:05.355378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:05.397227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd16f2b-136a-42b7-a566-d57d9eacc4a2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:05.435696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9f5e17-a94e-4be7-a4eb-6d89fc677d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:05.436403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9f5e17-a94e-4be7-a4eb-6d89fc677d39 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:05.436594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9f5e17-a94e-4be7-a4eb-6d89fc677d39 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:05.436885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9f5e17-a94e-4be7-a4eb-6d89fc677d39 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:05.437625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9f5e17-a94e-4be7-a4eb-6d89fc677d39 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:05.439239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9f5e17-a94e-4be7-a4eb-6d89fc677d39 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:05.470360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 567/1134] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:29:05 2022] POST /identity/v3/projects => generated 311 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:29:07.573125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776abdf5-7c45-4512-9ef4-44a96978fd09 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:07.573701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776abdf5-7c45-4512-9ef4-44a96978fd09 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:07.573701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776abdf5-7c45-4512-9ef4-44a96978fd09 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:07.575118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 568/1135] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:07.581893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec97b48-f92a-4fad-a7a2-2812cbcc49ba None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:07.582435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec97b48-f92a-4fad-a7a2-2812cbcc49ba None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:07.582435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec97b48-f92a-4fad-a7a2-2812cbcc49ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:07.606204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-6ec97b48-f92a-4fad-a7a2-2812cbcc49ba None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:07.607621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6ec97b48-f92a-4fad-a7a2-2812cbcc49ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:07.698705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 568/1136] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:08.001566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776abdf5-7c45-4512-9ef4-44a96978fd09 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:08.097777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:08.122099 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776abdf5-7c45-4512-9ef4-44a96978fd09 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:08.169034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05168a3c-c081-4dd1-84f0-50a6f6b32a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:08.169752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05168a3c-c081-4dd1-84f0-50a6f6b32a94 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:08.170037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05168a3c-c081-4dd1-84f0-50a6f6b32a94 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:08.170307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05168a3c-c081-4dd1-84f0-50a6f6b32a94 None admin] PATH_INFO: `/v3/domains/default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:08.171490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05168a3c-c081-4dd1-84f0-50a6f6b32a94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:08.175550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05168a3c-c081-4dd1-84f0-50a6f6b32a94 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:08.177241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 569/1137] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:29:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:08.183430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec97b48-f92a-4fad-a7a2-2812cbcc49ba None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:08.243566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:08.309305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec97b48-f92a-4fad-a7a2-2812cbcc49ba None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:08.362108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd69e68-c8e6-42bd-9da3-0b90f2a59d63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:08.363030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd69e68-c8e6-42bd-9da3-0b90f2a59d63 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:08.363384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd69e68-c8e6-42bd-9da3-0b90f2a59d63 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:08.363739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd69e68-c8e6-42bd-9da3-0b90f2a59d63 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:08.364501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd69e68-c8e6-42bd-9da3-0b90f2a59d63 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:08.366416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd69e68-c8e6-42bd-9da3-0b90f2a59d63 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:08.400112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 569/1138] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:29:08 2022] POST /identity/v3/users => generated 286 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:29:10.456265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dfe7f8-0f9b-41e3-acea-4a0a77c4cbb4 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:10.456818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dfe7f8-0f9b-41e3-acea-4a0a77c4cbb4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:10.456818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dfe7f8-0f9b-41e3-acea-4a0a77c4cbb4 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:10.458121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 570/1139] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:10.465467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d428bbb-5ebf-4a26-bd1b-ffe01d9de657 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:10.465979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d428bbb-5ebf-4a26-bd1b-ffe01d9de657 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:10.465979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d428bbb-5ebf-4a26-bd1b-ffe01d9de657 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:10.486381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9d428bbb-5ebf-4a26-bd1b-ffe01d9de657 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:10.487678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9d428bbb-5ebf-4a26-bd1b-ffe01d9de657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:10.580646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 570/1140] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:10.857700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dfe7f8-0f9b-41e3-acea-4a0a77c4cbb4 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:10.986834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dfe7f8-0f9b-41e3-acea-4a0a77c4cbb4 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:11.023910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acc7ef3-91bd-4648-8dfa-252c93e4f42b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:11.024555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc7ef3-91bd-4648-8dfa-252c93e4f42b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:11.024836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc7ef3-91bd-4648-8dfa-252c93e4f42b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:11.025151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc7ef3-91bd-4648-8dfa-252c93e4f42b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:11.037695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acc7ef3-91bd-4648-8dfa-252c93e4f42b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-1acc7ef3-91bd-4648-8dfa-252c93e4f42b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:29:11.042999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:29:11.044307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:29:11.045260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 571/1141] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:29:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:29:11.054976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d428bbb-5ebf-4a26-bd1b-ffe01d9de657 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:11.091994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:11.176179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d428bbb-5ebf-4a26-bd1b-ffe01d9de657 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:11.216712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:11.217326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:11.217603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:11.217874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:11.218544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:11.218879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:11.220914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:11.229700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 571/1142] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:29:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:11.236248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acc7ef3-91bd-4648-8dfa-252c93e4f42b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:11.248554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:11.356807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acc7ef3-91bd-4648-8dfa-252c93e4f42b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:11.396046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49909573-e61d-462d-9e3e-270f2ee01e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:11.396671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49909573-e61d-462d-9e3e-270f2ee01e4e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:11.396950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49909573-e61d-462d-9e3e-270f2ee01e4e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:11.397217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49909573-e61d-462d-9e3e-270f2ee01e4e None admin] PATH_INFO: `/v3/users/957f1978b4ac4609bc62c579cc0a504e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:11.411899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49909573-e61d-462d-9e3e-270f2ee01e4e None admin] RBAC: Authorizing `identity:get_user(user_id=957f1978b4ac4609bc62c579cc0a504e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:11.413360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49909573-e61d-462d-9e3e-270f2ee01e4e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:11.415106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 572/1143] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:29:11 2022] GET /identity/v3/users/957f1978b4ac4609bc62c579cc0a504e => generated 286 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:11.421838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:11.526391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:11.536308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9deb58ec-cf0d-4c9b-a68e-483014426963 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:11.583359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f30be6-dfad-4329-8e18-48f5f7c35b71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:11.583966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f30be6-dfad-4329-8e18-48f5f7c35b71 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:11.584247 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f30be6-dfad-4329-8e18-48f5f7c35b71 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:11.584517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f30be6-dfad-4329-8e18-48f5f7c35b71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:11.679250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f30be6-dfad-4329-8e18-48f5f7c35b71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:11.680712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f30be6-dfad-4329-8e18-48f5f7c35b71 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:11.794537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:11.804512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 572/1144] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:29:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:29:11.810512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49909573-e61d-462d-9e3e-270f2ee01e4e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:11.896825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:11.939946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49909573-e61d-462d-9e3e-270f2ee01e4e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:11.978845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5b2f5a-561a-4dda-8e4b-fb67611c15d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:11.979473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5b2f5a-561a-4dda-8e4b-fb67611c15d4 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:11.979750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5b2f5a-561a-4dda-8e4b-fb67611c15d4 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:11.980021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5b2f5a-561a-4dda-8e4b-fb67611c15d4 None admin] PATH_INFO: `/v3/projects/19cec7435c424ee68ddc760b7b6c97d0` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:11.981158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5b2f5a-561a-4dda-8e4b-fb67611c15d4 None admin] RBAC: Authorizing `identity:get_project(project_id=19cec7435c424ee68ddc760b7b6c97d0)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:11.982596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5b2f5a-561a-4dda-8e4b-fb67611c15d4 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:11.984274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 573/1145] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:29:11 2022] GET /identity/v3/projects/19cec7435c424ee68ddc760b7b6c97d0 => generated 311 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:11.991860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f30be6-dfad-4329-8e18-48f5f7c35b71 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:12.112155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f30be6-dfad-4329-8e18-48f5f7c35b71 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:12.159134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8783db70-a2a6-4703-944f-4013071f8eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:12.159916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8783db70-a2a6-4703-944f-4013071f8eb5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:12.160242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8783db70-a2a6-4703-944f-4013071f8eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:12.160553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8783db70-a2a6-4703-944f-4013071f8eb5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:12.161367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8783db70-a2a6-4703-944f-4013071f8eb5 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=19cec7435c424ee68ddc760b7b6c97d0, user.id=957f1978b4ac4609bc62c579cc0a504e) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:12.161732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8783db70-a2a6-4703-944f-4013071f8eb5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:12.163257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8783db70-a2a6-4703-944f-4013071f8eb5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:12.175065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 573/1146] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:29:11 2022] GET /identity/v3/role_assignments?user.id=957f1978b4ac4609bc62c579cc0a504e&scope.project.id=19cec7435c424ee68ddc760b7b6c97d0&role.id=38171c89968943bd92d9e9758db65615 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:14.148703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ef8a52-a714-4731-b2ce-39f67093656d None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:14.149439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ef8a52-a714-4731-b2ce-39f67093656d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:14.149439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ef8a52-a714-4731-b2ce-39f67093656d None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:14.150598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 574/1147] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:14.156511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d2adeb-3a9d-4ac1-865b-c66670d416af None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:14.156968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d2adeb-3a9d-4ac1-865b-c66670d416af None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:14.157436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d2adeb-3a9d-4ac1-865b-c66670d416af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:14.177765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-32d2adeb-3a9d-4ac1-865b-c66670d416af None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:14.179148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-32d2adeb-3a9d-4ac1-865b-c66670d416af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:14.237122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:14.254612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 574/1148] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:14.529494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ef8a52-a714-4731-b2ce-39f67093656d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:14.654991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ef8a52-a714-4731-b2ce-39f67093656d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:14.698225 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32318370-4154-43a6-bce6-6766bf66b500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:14.698885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32318370-4154-43a6-bce6-6766bf66b500 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:14.699171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32318370-4154-43a6-bce6-6766bf66b500 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:14.699442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32318370-4154-43a6-bce6-6766bf66b500 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:14.701413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:14.711604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32318370-4154-43a6-bce6-6766bf66b500 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-32318370-4154-43a6-bce6-6766bf66b500 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:29:14.716593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:29:14.718541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:29:14.721149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 575/1149] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:29:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:29:14.725374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d2adeb-3a9d-4ac1-865b-c66670d416af None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:14.857470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d2adeb-3a9d-4ac1-865b-c66670d416af None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:14.895664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:14.896623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:14.896906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:14.897177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:14.897890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:14.898194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:14.899790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:14.909670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 575/1150] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:29:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:14.916144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32318370-4154-43a6-bce6-6766bf66b500 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:15.039988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32318370-4154-43a6-bce6-6766bf66b500 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:15.080504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1575af-7b50-4488-aaf6-a4dda08e4811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:15.081205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1575af-7b50-4488-aaf6-a4dda08e4811 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:15.081491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1575af-7b50-4488-aaf6-a4dda08e4811 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:15.081805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1575af-7b50-4488-aaf6-a4dda08e4811 None admin] PATH_INFO: `/v3/users/957f1978b4ac4609bc62c579cc0a504e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:15.083102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1575af-7b50-4488-aaf6-a4dda08e4811 None admin] RBAC: Authorizing `identity:get_user(user_id=957f1978b4ac4609bc62c579cc0a504e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:15.084680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1575af-7b50-4488-aaf6-a4dda08e4811 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:15.086488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 576/1151] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:29:14 2022] GET /identity/v3/users/957f1978b4ac4609bc62c579cc0a504e => generated 286 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:15.094097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:15.154184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:15.229277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d8fac-cbc8-44e0-b94e-d53ec290d30d None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:15.266645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5fdc55-49a8-44c0-a128-6ff70980027a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:15.267814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5fdc55-49a8-44c0-a128-6ff70980027a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:15.268092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5fdc55-49a8-44c0-a128-6ff70980027a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:15.268368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5fdc55-49a8-44c0-a128-6ff70980027a None admin] PATH_INFO: `/v3/projects/19cec7435c424ee68ddc760b7b6c97d0` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:15.269518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5fdc55-49a8-44c0-a128-6ff70980027a None admin] RBAC: Authorizing `identity:get_project(project_id=19cec7435c424ee68ddc760b7b6c97d0)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:15.271057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5fdc55-49a8-44c0-a128-6ff70980027a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:15.272787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 576/1152] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:29:15 2022] GET /identity/v3/projects/19cec7435c424ee68ddc760b7b6c97d0 => generated 311 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:15.279328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1575af-7b50-4488-aaf6-a4dda08e4811 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:15.353236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:15.408066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1575af-7b50-4488-aaf6-a4dda08e4811 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:15.449293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92418901-aa1f-42e7-a499-5860fa5b6a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:15.449960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92418901-aa1f-42e7-a499-5860fa5b6a74 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:15.450235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92418901-aa1f-42e7-a499-5860fa5b6a74 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:15.450511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92418901-aa1f-42e7-a499-5860fa5b6a74 None admin] PATH_INFO: `/v3/projects/19cec7435c424ee68ddc760b7b6c97d0/users/957f1978b4ac4609bc62c579cc0a504e/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:15.452574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92418901-aa1f-42e7-a499-5860fa5b6a74 None admin] RBAC: Authorizing `identity:create_grant(project_id=19cec7435c424ee68ddc760b7b6c97d0, user_id=957f1978b4ac4609bc62c579cc0a504e, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:15.454021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92418901-aa1f-42e7-a499-5860fa5b6a74 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:15.468790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 577/1153] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:29:15 2022] PUT /identity/v3/projects/19cec7435c424ee68ddc760b7b6c97d0/users/957f1978b4ac4609bc62c579cc0a504e/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:29:17.396615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97936c31-7d2e-4d0c-a56d-162d770309b4 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:17.397155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97936c31-7d2e-4d0c-a56d-162d770309b4 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:17.397155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97936c31-7d2e-4d0c-a56d-162d770309b4 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:17.398497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 577/1154] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:17.404778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd700c7-448c-4b82-9aa1-1107a2d3cee0 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:17.405365 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd700c7-448c-4b82-9aa1-1107a2d3cee0 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:17.405889 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd700c7-448c-4b82-9aa1-1107a2d3cee0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:17.428824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8dd700c7-448c-4b82-9aa1-1107a2d3cee0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:17.430027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8dd700c7-448c-4b82-9aa1-1107a2d3cee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:17.504315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 578/1155] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:29:17.907116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97936c31-7d2e-4d0c-a56d-162d770309b4 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:18.049616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97936c31-7d2e-4d0c-a56d-162d770309b4 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:18.067836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:18.088872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df6616f-fc09-4651-9e92-f52e9e0bb958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:18.089415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df6616f-fc09-4651-9e92-f52e9e0bb958 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:18.089691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df6616f-fc09-4651-9e92-f52e9e0bb958 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:18.089958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df6616f-fc09-4651-9e92-f52e9e0bb958 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:18.101799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df6616f-fc09-4651-9e92-f52e9e0bb958 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-2df6616f-fc09-4651-9e92-f52e9e0bb958 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:29:18.106768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:29:18.107899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:29:18.109159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 578/1156] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:29:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:29:18.114340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd700c7-448c-4b82-9aa1-1107a2d3cee0 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:18.186094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:18.240399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd700c7-448c-4b82-9aa1-1107a2d3cee0 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:18.277457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:18.278075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:18.278355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:18.278627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:18.279360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:18.279665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:18.281047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:18.290446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 579/1157] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:29:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:29:18.297062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df6616f-fc09-4651-9e92-f52e9e0bb958 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:18.414264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df6616f-fc09-4651-9e92-f52e9e0bb958 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:18.465110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88469368-0aca-4987-b480-a5cd1db2214e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:18.465746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88469368-0aca-4987-b480-a5cd1db2214e None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:18.466022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88469368-0aca-4987-b480-a5cd1db2214e None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:18.466287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88469368-0aca-4987-b480-a5cd1db2214e None admin] PATH_INFO: `/v3/users/957f1978b4ac4609bc62c579cc0a504e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:18.467462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88469368-0aca-4987-b480-a5cd1db2214e None admin] RBAC: Authorizing `identity:get_user(user_id=957f1978b4ac4609bc62c579cc0a504e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:18.468896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88469368-0aca-4987-b480-a5cd1db2214e None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:18.470484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 579/1158] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:29:18 2022] GET /identity/v3/users/957f1978b4ac4609bc62c579cc0a504e => generated 286 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:18.476983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:18.589018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d422080-9ca3-426e-bd46-eccb30a7f880 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:18.607240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:18.632415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73437a-c1e3-4c1d-ab1b-c33cc97c031c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:18.633028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73437a-c1e3-4c1d-ab1b-c33cc97c031c None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:18.633304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73437a-c1e3-4c1d-ab1b-c33cc97c031c None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:18.633572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73437a-c1e3-4c1d-ab1b-c33cc97c031c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:18.727263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de73437a-c1e3-4c1d-ab1b-c33cc97c031c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:18.729701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de73437a-c1e3-4c1d-ab1b-c33cc97c031c None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:18.865206 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 580/1159] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:29:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:29:18.872340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88469368-0aca-4987-b480-a5cd1db2214e None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:18.922464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:18.985646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88469368-0aca-4987-b480-a5cd1db2214e None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:19.027862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f4aa6-5beb-4dd6-bdd3-8485384fca34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:19.028575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f4aa6-5beb-4dd6-bdd3-8485384fca34 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:19.028865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f4aa6-5beb-4dd6-bdd3-8485384fca34 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:19.029220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f4aa6-5beb-4dd6-bdd3-8485384fca34 None admin] PATH_INFO: `/v3/projects/19cec7435c424ee68ddc760b7b6c97d0` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:19.030406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018f4aa6-5beb-4dd6-bdd3-8485384fca34 None admin] RBAC: Authorizing `identity:get_project(project_id=19cec7435c424ee68ddc760b7b6c97d0)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:19.031911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018f4aa6-5beb-4dd6-bdd3-8485384fca34 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:19.033625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 580/1160] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:29:18 2022] GET /identity/v3/projects/19cec7435c424ee68ddc760b7b6c97d0 => generated 311 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:19.042143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73437a-c1e3-4c1d-ab1b-c33cc97c031c None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:19.066486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:19.186509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73437a-c1e3-4c1d-ab1b-c33cc97c031c None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:19.235212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e4c84b-2b94-4c8c-95da-c19a4ac7cee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:19.236103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e4c84b-2b94-4c8c-95da-c19a4ac7cee5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:19.236558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e4c84b-2b94-4c8c-95da-c19a4ac7cee5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:19.237072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e4c84b-2b94-4c8c-95da-c19a4ac7cee5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:19.237886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e4c84b-2b94-4c8c-95da-c19a4ac7cee5 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=19cec7435c424ee68ddc760b7b6c97d0, user.id=957f1978b4ac4609bc62c579cc0a504e) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:19.238288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e4c84b-2b94-4c8c-95da-c19a4ac7cee5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:19.239932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e4c84b-2b94-4c8c-95da-c19a4ac7cee5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:19.247816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 581/1161] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:29:19 2022] GET /identity/v3/role_assignments?user.id=957f1978b4ac4609bc62c579cc0a504e&scope.project.id=19cec7435c424ee68ddc760b7b6c97d0&role.id=38171c89968943bd92d9e9758db65615 => generated 608 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:21.264157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07bced-7115-409c-a85f-f91225c30c74 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:21.264743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07bced-7115-409c-a85f-f91225c30c74 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:21.264918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07bced-7115-409c-a85f-f91225c30c74 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:21.266713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 581/1162] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:29:21.273321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c47896-0cc9-41e6-8f94-73d213ba8fd3 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:21.274028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c47896-0cc9-41e6-8f94-73d213ba8fd3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:21.274028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c47896-0cc9-41e6-8f94-73d213ba8fd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:21.294848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-71c47896-0cc9-41e6-8f94-73d213ba8fd3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:21.296066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-71c47896-0cc9-41e6-8f94-73d213ba8fd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:21.389602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 582/1163] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:29:21.686274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c07bced-7115-409c-a85f-f91225c30c74 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:21.814220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c07bced-7115-409c-a85f-f91225c30c74 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:21.833252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:21.865895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8412a9-8c8b-4dae-8e0f-0ba0c827ae4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:21.866938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8412a9-8c8b-4dae-8e0f-0ba0c827ae4a None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:21.867431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8412a9-8c8b-4dae-8e0f-0ba0c827ae4a None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:21.867882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8412a9-8c8b-4dae-8e0f-0ba0c827ae4a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:21.940806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8412a9-8c8b-4dae-8e0f-0ba0c827ae4a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:21.942191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8412a9-8c8b-4dae-8e0f-0ba0c827ae4a None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:22.053122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 582/1164] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:29:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:29:22.061557 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c47896-0cc9-41e6-8f94-73d213ba8fd3 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:22.080200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:22.178251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c47896-0cc9-41e6-8f94-73d213ba8fd3 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:22.217318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8613bb7-9eda-451e-8904-e94189346492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:22.217848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8613bb7-9eda-451e-8904-e94189346492 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:22.218128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8613bb7-9eda-451e-8904-e94189346492 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:22.218408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8613bb7-9eda-451e-8904-e94189346492 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:22.225408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8613bb7-9eda-451e-8904-e94189346492 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:22.226896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8613bb7-9eda-451e-8904-e94189346492 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-d8613bb7-9eda-451e-8904-e94189346492 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:29:22.238317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:29:22.239756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:29:22.240838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 583/1165] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:29:22 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:29:22.248583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8412a9-8c8b-4dae-8e0f-0ba0c827ae4a None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:22.265706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:22.359544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8412a9-8c8b-4dae-8e0f-0ba0c827ae4a None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:22.409954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412340cc-1dd8-4da9-852f-43948a0ea8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:22.410569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412340cc-1dd8-4da9-852f-43948a0ea8f3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:22.410869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412340cc-1dd8-4da9-852f-43948a0ea8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:22.411140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412340cc-1dd8-4da9-852f-43948a0ea8f3 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:22.411793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412340cc-1dd8-4da9-852f-43948a0ea8f3 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:22.412087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412340cc-1dd8-4da9-852f-43948a0ea8f3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:22.413457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412340cc-1dd8-4da9-852f-43948a0ea8f3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:22.421753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 583/1166] 10.222.0.77 () {56 vars in 1115 bytes} [Mon Aug 29 22:29:22 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:24.388366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900605-d1c4-46b2-b3fd-daf7fd3dd546 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:24.389019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900605-d1c4-46b2-b3fd-daf7fd3dd546 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:24.389019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900605-d1c4-46b2-b3fd-daf7fd3dd546 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:24.390231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 584/1167] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:24.395592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb73118-55ff-4e3c-99f7-8e45ef1712bf None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:24.396195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb73118-55ff-4e3c-99f7-8e45ef1712bf None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:24.396195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb73118-55ff-4e3c-99f7-8e45ef1712bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:24.416213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2fb73118-55ff-4e3c-99f7-8e45ef1712bf None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:24.417757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2fb73118-55ff-4e3c-99f7-8e45ef1712bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:24.490093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 584/1168] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:24.763464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63900605-d1c4-46b2-b3fd-daf7fd3dd546 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:24.851194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:24.893435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63900605-d1c4-46b2-b3fd-daf7fd3dd546 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:24.930467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83be7503-c944-46c7-b912-2e4bb5a467b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:24.931164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be7503-c944-46c7-b912-2e4bb5a467b1 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:24.931440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be7503-c944-46c7-b912-2e4bb5a467b1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:24.931749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be7503-c944-46c7-b912-2e4bb5a467b1 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:24.932447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83be7503-c944-46c7-b912-2e4bb5a467b1 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:24.933825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83be7503-c944-46c7-b912-2e4bb5a467b1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:24.951788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 585/1169] 10.222.0.77 () {60 vars in 1138 bytes} [Mon Aug 29 22:29:24 2022] POST /identity/v3/domains => generated 267 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:29:26.949444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea163a7a-3a33-436f-b3d4-08dedfcdff00 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:26.950007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea163a7a-3a33-436f-b3d4-08dedfcdff00 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:26.950223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea163a7a-3a33-436f-b3d4-08dedfcdff00 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:26.951707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 585/1170] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:26.958047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3939846-478d-49ca-a1f4-8b508b405a36 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:26.958913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3939846-478d-49ca-a1f4-8b508b405a36 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:26.958913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3939846-478d-49ca-a1f4-8b508b405a36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:26.978215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f3939846-478d-49ca-a1f4-8b508b405a36 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:26.979477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f3939846-478d-49ca-a1f4-8b508b405a36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:27.058972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 586/1171] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:27.332886 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea163a7a-3a33-436f-b3d4-08dedfcdff00 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:27.343070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:27.477859 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea163a7a-3a33-436f-b3d4-08dedfcdff00 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:27.519995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f55bcf-1408-4a0e-a82a-8d900a6abf71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:27.520624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f55bcf-1408-4a0e-a82a-8d900a6abf71 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:27.520900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f55bcf-1408-4a0e-a82a-8d900a6abf71 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:27.521170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f55bcf-1408-4a0e-a82a-8d900a6abf71 None admin] PATH_INFO: `/v3/domains/b11def3e7b174425aa26af83adfc1090` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:27.532992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f55bcf-1408-4a0e-a82a-8d900a6abf71 None admin] RBAC: Authorizing `identity:get_domain(domain_id=b11def3e7b174425aa26af83adfc1090)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:27.534688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f55bcf-1408-4a0e-a82a-8d900a6abf71 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:27.536656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 586/1172] 10.222.0.77 () {56 vars in 1183 bytes} [Mon Aug 29 22:29:27 2022] GET /identity/v3/domains/b11def3e7b174425aa26af83adfc1090 => generated 267 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:29:27.543265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3939846-478d-49ca-a1f4-8b508b405a36 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:27.613400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:27.666218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3939846-478d-49ca-a1f4-8b508b405a36 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:27.710654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7299c2-3716-4de1-a45a-7f406edd7777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:27.711291 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7299c2-3716-4de1-a45a-7f406edd7777 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:27.711571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7299c2-3716-4de1-a45a-7f406edd7777 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:27.711846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7299c2-3716-4de1-a45a-7f406edd7777 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:27.712510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7299c2-3716-4de1-a45a-7f406edd7777 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:27.714389 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7299c2-3716-4de1-a45a-7f406edd7777 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:27.746392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 587/1173] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:29:27 2022] POST /identity/v3/projects => generated 361 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:29:29.774568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e671af58-b8f1-4fa3-ac2a-58e4638478a3 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:29.775249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e671af58-b8f1-4fa3-ac2a-58e4638478a3 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:29.775249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e671af58-b8f1-4fa3-ac2a-58e4638478a3 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:29.776490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 587/1174] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:29.790007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131d5ccc-4dbd-440d-a4ee-19d6e01e9415 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:29.790552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131d5ccc-4dbd-440d-a4ee-19d6e01e9415 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:29.790552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131d5ccc-4dbd-440d-a4ee-19d6e01e9415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:29.810258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-131d5ccc-4dbd-440d-a4ee-19d6e01e9415 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:29.811576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-131d5ccc-4dbd-440d-a4ee-19d6e01e9415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:29.897324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 588/1175] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:30.171429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e671af58-b8f1-4fa3-ac2a-58e4638478a3 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:30.263768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:30.307356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e671af58-b8f1-4fa3-ac2a-58e4638478a3 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:30.350180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13fc469-bc54-44b2-883e-17c834e7be8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:30.350835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13fc469-bc54-44b2-883e-17c834e7be8b None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:30.351116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13fc469-bc54-44b2-883e-17c834e7be8b None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:30.351389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13fc469-bc54-44b2-883e-17c834e7be8b None admin] PATH_INFO: `/v3/domains/b11def3e7b174425aa26af83adfc1090` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:30.352531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13fc469-bc54-44b2-883e-17c834e7be8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=b11def3e7b174425aa26af83adfc1090)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:30.353976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13fc469-bc54-44b2-883e-17c834e7be8b None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:30.355587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 588/1176] 10.222.0.77 () {56 vars in 1183 bytes} [Mon Aug 29 22:29:30 2022] GET /identity/v3/domains/b11def3e7b174425aa26af83adfc1090 => generated 267 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:30.361470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131d5ccc-4dbd-440d-a4ee-19d6e01e9415 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:30.407651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:30.471570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131d5ccc-4dbd-440d-a4ee-19d6e01e9415 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:30.508598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e8fa39-1cf1-419f-832a-9d5671041f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:30.509208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e8fa39-1cf1-419f-832a-9d5671041f18 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:30.509487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e8fa39-1cf1-419f-832a-9d5671041f18 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:30.509755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e8fa39-1cf1-419f-832a-9d5671041f18 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:30.510428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e8fa39-1cf1-419f-832a-9d5671041f18 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:30.511888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e8fa39-1cf1-419f-832a-9d5671041f18 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:30.534619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 589/1177] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:29:30 2022] POST /identity/v3/users => generated 311 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:29:32.563893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7221d857-c4a3-4656-8d4a-fe6765181771 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:32.564443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7221d857-c4a3-4656-8d4a-fe6765181771 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:32.564443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7221d857-c4a3-4656-8d4a-fe6765181771 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:32.566293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 589/1178] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:29:32.572158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89003836-6777-465e-ba69-6d617e155497 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:32.572965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89003836-6777-465e-ba69-6d617e155497 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:32.572965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89003836-6777-465e-ba69-6d617e155497 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:32.592386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-89003836-6777-465e-ba69-6d617e155497 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:32.593603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-89003836-6777-465e-ba69-6d617e155497 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:32.674250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 590/1179] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:32.965783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7221d857-c4a3-4656-8d4a-fe6765181771 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:33.107867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7221d857-c4a3-4656-8d4a-fe6765181771 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:33.151398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4c3f2e-2ed3-41d1-91cf-398f86a1b6a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:33.152213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4c3f2e-2ed3-41d1-91cf-398f86a1b6a5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:33.152689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4c3f2e-2ed3-41d1-91cf-398f86a1b6a5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:33.153040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4c3f2e-2ed3-41d1-91cf-398f86a1b6a5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:33.155798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:33.170018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4c3f2e-2ed3-41d1-91cf-398f86a1b6a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-de4c3f2e-2ed3-41d1-91cf-398f86a1b6a5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:29:33.179732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:29:33.181003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:33.182116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:29:33.186936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 590/1180] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:29:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:29:33.190939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89003836-6777-465e-ba69-6d617e155497 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:33.224009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:33.311676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89003836-6777-465e-ba69-6d617e155497 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:33.356126 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:33.357115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:33.357525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:33.357807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:33.359033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:33.359351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:33.361534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:33.374701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 591/1181] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:29:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:33.384601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4c3f2e-2ed3-41d1-91cf-398f86a1b6a5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:33.508129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4c3f2e-2ed3-41d1-91cf-398f86a1b6a5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:33.548655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d3d96-67f1-4ec0-bef4-3ca1d50674b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:33.549214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d3d96-67f1-4ec0-bef4-3ca1d50674b5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:33.549713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d3d96-67f1-4ec0-bef4-3ca1d50674b5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:33.550098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d3d96-67f1-4ec0-bef4-3ca1d50674b5 None admin] PATH_INFO: `/v3/users/50b4483df09c4072bf0a2c4c12f20ebd` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:33.568486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941d3d96-67f1-4ec0-bef4-3ca1d50674b5 None admin] RBAC: Authorizing `identity:get_user(user_id=50b4483df09c4072bf0a2c4c12f20ebd)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:33.570176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941d3d96-67f1-4ec0-bef4-3ca1d50674b5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:33.572213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 591/1182] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:29:33 2022] GET /identity/v3/users/50b4483df09c4072bf0a2c4c12f20ebd => generated 311 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:33.578725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:33.686058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:33.699767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5b5832-49da-473f-a197-7a1f1082cfb9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:33.738443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919cbc29-6a0d-45bb-a731-65a889393ffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:33.739089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919cbc29-6a0d-45bb-a731-65a889393ffe None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:33.739374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919cbc29-6a0d-45bb-a731-65a889393ffe None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:33.739648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919cbc29-6a0d-45bb-a731-65a889393ffe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:33.835886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919cbc29-6a0d-45bb-a731-65a889393ffe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:33.837529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919cbc29-6a0d-45bb-a731-65a889393ffe None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:33.950067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:33.965343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 592/1183] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:29:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:29:33.974536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d3d96-67f1-4ec0-bef4-3ca1d50674b5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:34.051533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:34.115719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d3d96-67f1-4ec0-bef4-3ca1d50674b5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:34.162966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9598f9c2-e0e6-4d93-8d1e-b8a2473ff2d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:34.163732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598f9c2-e0e6-4d93-8d1e-b8a2473ff2d6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:34.164069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598f9c2-e0e6-4d93-8d1e-b8a2473ff2d6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:34.164393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598f9c2-e0e6-4d93-8d1e-b8a2473ff2d6 None admin] PATH_INFO: `/v3/projects/68b675bec1504360ab103cbed62f3f13` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:34.165721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9598f9c2-e0e6-4d93-8d1e-b8a2473ff2d6 None admin] RBAC: Authorizing `identity:get_project(project_id=68b675bec1504360ab103cbed62f3f13)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:34.167601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9598f9c2-e0e6-4d93-8d1e-b8a2473ff2d6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:34.169427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 592/1184] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:29:33 2022] GET /identity/v3/projects/68b675bec1504360ab103cbed62f3f13 => generated 361 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:34.176022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919cbc29-6a0d-45bb-a731-65a889393ffe None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:34.292660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919cbc29-6a0d-45bb-a731-65a889393ffe None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:34.338599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abc2e0f-2b10-45d9-9a52-26c5986e327b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:34.339235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abc2e0f-2b10-45d9-9a52-26c5986e327b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:34.339512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abc2e0f-2b10-45d9-9a52-26c5986e327b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:34.339804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abc2e0f-2b10-45d9-9a52-26c5986e327b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:34.340547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abc2e0f-2b10-45d9-9a52-26c5986e327b None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=68b675bec1504360ab103cbed62f3f13, user.id=50b4483df09c4072bf0a2c4c12f20ebd) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:34.340856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abc2e0f-2b10-45d9-9a52-26c5986e327b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:34.342301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abc2e0f-2b10-45d9-9a52-26c5986e327b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:34.349380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 593/1185] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:29:34 2022] GET /identity/v3/role_assignments?user.id=50b4483df09c4072bf0a2c4c12f20ebd&scope.project.id=68b675bec1504360ab103cbed62f3f13&role.id=38171c89968943bd92d9e9758db65615 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:29:36.449919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e468e8-a431-4cbf-a99c-c357bd3d180b None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:36.450537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e468e8-a431-4cbf-a99c-c357bd3d180b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:36.450954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e468e8-a431-4cbf-a99c-c357bd3d180b None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:36.453463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 593/1186] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:36.459639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4457d6c-d18a-4097-ad01-14017883fa7b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:36.460503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4457d6c-d18a-4097-ad01-14017883fa7b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:36.460503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4457d6c-d18a-4097-ad01-14017883fa7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:36.486540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e4457d6c-d18a-4097-ad01-14017883fa7b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:36.489261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e4457d6c-d18a-4097-ad01-14017883fa7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:36.564462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:36.592353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 594/1187] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:36.891373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e468e8-a431-4cbf-a99c-c357bd3d180b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:37.014528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e468e8-a431-4cbf-a99c-c357bd3d180b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:37.049093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:37.068910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfeccc-3138-4dc3-a2c2-cf74b25b347f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:37.069548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfeccc-3138-4dc3-a2c2-cf74b25b347f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:37.069829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfeccc-3138-4dc3-a2c2-cf74b25b347f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:37.070102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfeccc-3138-4dc3-a2c2-cf74b25b347f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:37.081475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfeccc-3138-4dc3-a2c2-cf74b25b347f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-06cfeccc-3138-4dc3-a2c2-cf74b25b347f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:29:37.086306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:29:37.088348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:29:37.090048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 594/1188] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:29:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:29:37.093742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4457d6c-d18a-4097-ad01-14017883fa7b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:37.223023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4457d6c-d18a-4097-ad01-14017883fa7b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:37.271785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:37.272712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:37.273172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:37.273629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:37.274715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:37.275231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:37.277530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:37.293371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 595/1189] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:29:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:29:37.300087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfeccc-3138-4dc3-a2c2-cf74b25b347f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:37.435021 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfeccc-3138-4dc3-a2c2-cf74b25b347f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:37.494418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c392c975-f7c3-4746-b204-7ae12d364b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:37.495465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c392c975-f7c3-4746-b204-7ae12d364b46 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:37.495874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c392c975-f7c3-4746-b204-7ae12d364b46 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:37.496356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c392c975-f7c3-4746-b204-7ae12d364b46 None admin] PATH_INFO: `/v3/users/50b4483df09c4072bf0a2c4c12f20ebd` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:37.498397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c392c975-f7c3-4746-b204-7ae12d364b46 None admin] RBAC: Authorizing `identity:get_user(user_id=50b4483df09c4072bf0a2c4c12f20ebd)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:37.501197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c392c975-f7c3-4746-b204-7ae12d364b46 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:37.504273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 595/1190] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:29:37 2022] GET /identity/v3/users/50b4483df09c4072bf0a2c4c12f20ebd => generated 311 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:29:37.512906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:37.575825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:37.655740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdbfb7-5cd6-4d54-ad69-168cb7d1ee23 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:37.693884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a30092a-8c98-4d17-ac1c-e1dfdc925347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:37.694436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30092a-8c98-4d17-ac1c-e1dfdc925347 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:37.694715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30092a-8c98-4d17-ac1c-e1dfdc925347 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:37.695024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30092a-8c98-4d17-ac1c-e1dfdc925347 None admin] PATH_INFO: `/v3/projects/68b675bec1504360ab103cbed62f3f13` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:37.696174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a30092a-8c98-4d17-ac1c-e1dfdc925347 None admin] RBAC: Authorizing `identity:get_project(project_id=68b675bec1504360ab103cbed62f3f13)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:37.697642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a30092a-8c98-4d17-ac1c-e1dfdc925347 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:37.699373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 596/1191] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:29:37 2022] GET /identity/v3/projects/68b675bec1504360ab103cbed62f3f13 => generated 361 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:37.706321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c392c975-f7c3-4746-b204-7ae12d364b46 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:37.762424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:37.839641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c392c975-f7c3-4746-b204-7ae12d364b46 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:37.900876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd21c493-3003-4b29-ad8e-8a0a0bf192fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:37.902043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21c493-3003-4b29-ad8e-8a0a0bf192fd None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:37.902548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21c493-3003-4b29-ad8e-8a0a0bf192fd None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:37.903068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21c493-3003-4b29-ad8e-8a0a0bf192fd None admin] PATH_INFO: `/v3/projects/68b675bec1504360ab103cbed62f3f13/users/50b4483df09c4072bf0a2c4c12f20ebd/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:37.906628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd21c493-3003-4b29-ad8e-8a0a0bf192fd None admin] RBAC: Authorizing `identity:create_grant(project_id=68b675bec1504360ab103cbed62f3f13, user_id=50b4483df09c4072bf0a2c4c12f20ebd, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:37.909435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd21c493-3003-4b29-ad8e-8a0a0bf192fd None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:37.931022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 596/1192] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:29:37 2022] PUT /identity/v3/projects/68b675bec1504360ab103cbed62f3f13/users/50b4483df09c4072bf0a2c4c12f20ebd/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:29:39.978824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d42df7-0b68-4f08-a3c4-7909280c9374 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:39.980123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d42df7-0b68-4f08-a3c4-7909280c9374 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:39.980123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d42df7-0b68-4f08-a3c4-7909280c9374 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:39.980807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 597/1193] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:29:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:29:39.988213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580eb183-0e04-459d-947c-4cd09328207b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:39.988707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580eb183-0e04-459d-947c-4cd09328207b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:39.989045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580eb183-0e04-459d-947c-4cd09328207b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:40.007402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-580eb183-0e04-459d-947c-4cd09328207b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:29:40.008621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-580eb183-0e04-459d-947c-4cd09328207b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:29:40.083152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 597/1194] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:29:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:29:40.356510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d42df7-0b68-4f08-a3c4-7909280c9374 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:40.497310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d42df7-0b68-4f08-a3c4-7909280c9374 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:40.512679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:40.537911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618e5baf-7f1a-482e-af6e-9b7ae3f34ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:40.538545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e5baf-7f1a-482e-af6e-9b7ae3f34ad9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:40.538857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e5baf-7f1a-482e-af6e-9b7ae3f34ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:40.539128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e5baf-7f1a-482e-af6e-9b7ae3f34ad9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:40.553271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618e5baf-7f1a-482e-af6e-9b7ae3f34ad9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-618e5baf-7f1a-482e-af6e-9b7ae3f34ad9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:29:40.558443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:29:40.560149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:29:40.561954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 598/1195] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:29:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:29:40.569287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580eb183-0e04-459d-947c-4cd09328207b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:40.614618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:40.682015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580eb183-0e04-459d-947c-4cd09328207b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:40.730484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:40.731546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:40.732013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:40.732478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:40.733591 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:40.734082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:40.736417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:40.751088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 598/1196] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:29:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:29:40.758194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618e5baf-7f1a-482e-af6e-9b7ae3f34ad9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:40.893127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618e5baf-7f1a-482e-af6e-9b7ae3f34ad9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:40.948303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c08b6a-4294-4d17-ab20-1896d92336e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:40.949282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c08b6a-4294-4d17-ab20-1896d92336e2 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:40.949578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c08b6a-4294-4d17-ab20-1896d92336e2 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:40.949859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c08b6a-4294-4d17-ab20-1896d92336e2 None admin] PATH_INFO: `/v3/users/50b4483df09c4072bf0a2c4c12f20ebd` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:40.951147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c08b6a-4294-4d17-ab20-1896d92336e2 None admin] RBAC: Authorizing `identity:get_user(user_id=50b4483df09c4072bf0a2c4c12f20ebd)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:40.952607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c08b6a-4294-4d17-ab20-1896d92336e2 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:40.954231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 599/1197] 10.222.0.77 () {56 vars in 1177 bytes} [Mon Aug 29 22:29:40 2022] GET /identity/v3/users/50b4483df09c4072bf0a2c4c12f20ebd => generated 311 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:29:40.961526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:41.098072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:41.102047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d59dcc6-f028-4ecc-a4d1-d245a53c0328 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:41.139024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a355ac02-0a91-431d-88a9-e22e8d94fd42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:41.139629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a355ac02-0a91-431d-88a9-e22e8d94fd42 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:41.139897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a355ac02-0a91-431d-88a9-e22e8d94fd42 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:41.140171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a355ac02-0a91-431d-88a9-e22e8d94fd42 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:41.234839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a355ac02-0a91-431d-88a9-e22e8d94fd42 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:41.236291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a355ac02-0a91-431d-88a9-e22e8d94fd42 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:41.367348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:41.374032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 599/1198] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:29:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:29:41.381393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c08b6a-4294-4d17-ab20-1896d92336e2 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:41.444480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:41.524885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c08b6a-4294-4d17-ab20-1896d92336e2 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:41.566302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6675e564-6e47-4b5b-b32e-eb60b9c46bb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:41.566974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675e564-6e47-4b5b-b32e-eb60b9c46bb7 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:41.567253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675e564-6e47-4b5b-b32e-eb60b9c46bb7 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:41.567516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675e564-6e47-4b5b-b32e-eb60b9c46bb7 None admin] PATH_INFO: `/v3/projects/68b675bec1504360ab103cbed62f3f13` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:41.568679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6675e564-6e47-4b5b-b32e-eb60b9c46bb7 None admin] RBAC: Authorizing `identity:get_project(project_id=68b675bec1504360ab103cbed62f3f13)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:41.570141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6675e564-6e47-4b5b-b32e-eb60b9c46bb7 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:41.572273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 600/1199] 10.222.0.77 () {56 vars in 1186 bytes} [Mon Aug 29 22:29:41 2022] GET /identity/v3/projects/68b675bec1504360ab103cbed62f3f13 => generated 361 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:41.578835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a355ac02-0a91-431d-88a9-e22e8d94fd42 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:29:41.697081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a355ac02-0a91-431d-88a9-e22e8d94fd42 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:29:41.744817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f930b72-5077-48b4-8196-7e6b62fac786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:29:41.745531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f930b72-5077-48b4-8196-7e6b62fac786 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:29:41.745998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f930b72-5077-48b4-8196-7e6b62fac786 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:29:41.746397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f930b72-5077-48b4-8196-7e6b62fac786 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:29:41.747466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f930b72-5077-48b4-8196-7e6b62fac786 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=68b675bec1504360ab103cbed62f3f13, user.id=50b4483df09c4072bf0a2c4c12f20ebd) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:29:41.747820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f930b72-5077-48b4-8196-7e6b62fac786 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:29:41.749641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f930b72-5077-48b4-8196-7e6b62fac786 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:29:41.758140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 600/1200] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:29:41 2022] GET /identity/v3/role_assignments?user.id=50b4483df09c4072bf0a2c4c12f20ebd&scope.project.id=68b675bec1504360ab103cbed62f3f13&role.id=38171c89968943bd92d9e9758db65615 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:29:51.561203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=46 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:29:51.754578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=68 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:46.204981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea57d858-02e4-423b-9223-ac5f4e4be1ad None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:46.205707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea57d858-02e4-423b-9223-ac5f4e4be1ad None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:46.205707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea57d858-02e4-423b-9223-ac5f4e4be1ad None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:46.207125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 601/1201] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:30:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:30:46.213523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a7bd3-4af1-432b-aed0-150f972a72e0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:46.214108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a7bd3-4af1-432b-aed0-150f972a72e0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:46.214108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a7bd3-4af1-432b-aed0-150f972a72e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:46.246022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c54a7bd3-4af1-432b-aed0-150f972a72e0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:30:46.248537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c54a7bd3-4af1-432b-aed0-150f972a72e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:30:46.390903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 601/1202] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:30:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:30:46.747020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea57d858-02e4-423b-9223-ac5f4e4be1ad None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:46.982583 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea57d858-02e4-423b-9223-ac5f4e4be1ad None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:47.047995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:47.049018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:47.049018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:47.049977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:47.070774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:47.072447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:30:47.102462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:47.104542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:30:47.106214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:30:47.106214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:47.106214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:30:47.109076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 602/1203] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:30:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:47.110859 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a7bd3-4af1-432b-aed0-150f972a72e0 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:47.353183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a7bd3-4af1-432b-aed0-150f972a72e0 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:47.408132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:47.430752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d40cc0-552d-418a-ad0b-47736af39f53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:47.431512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d40cc0-552d-418a-ad0b-47736af39f53 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:47.431871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d40cc0-552d-418a-ad0b-47736af39f53 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:47.432382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d40cc0-552d-418a-ad0b-47736af39f53 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:47.433145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d40cc0-552d-418a-ad0b-47736af39f53 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:47.433555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d40cc0-552d-418a-ad0b-47736af39f53 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:47.436047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d40cc0-552d-418a-ad0b-47736af39f53 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:47.466109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 602/1204] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:30:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:30:47.467532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:47.556672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:47.654250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1090f7-55ad-4ac5-bc7e-4824f5abd7de None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:47.713986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eaaa620-12f1-425d-a071-f27001e2f5ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:47.715009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaaa620-12f1-425d-a071-f27001e2f5ef None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:47.715393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaaa620-12f1-425d-a071-f27001e2f5ef None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:47.715766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaaa620-12f1-425d-a071-f27001e2f5ef None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:47.716665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eaaa620-12f1-425d-a071-f27001e2f5ef None admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:47.718366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eaaa620-12f1-425d-a071-f27001e2f5ef None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:47.758708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 603/1205] 10.222.0.77 () {60 vars in 1132 bytes} [Mon Aug 29 22:30:47 2022] POST /identity/v3/users => generated 251 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:30:50.898091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06f71e-28cd-4d88-8c2f-86f6a8c05cc9 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:50.899026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06f71e-28cd-4d88-8c2f-86f6a8c05cc9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:50.899026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06f71e-28cd-4d88-8c2f-86f6a8c05cc9 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:50.900411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 603/1206] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:30:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:30:50.907665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c0e089-ca57-4c60-a6b7-ca89feffe3cb None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:50.908616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c0e089-ca57-4c60-a6b7-ca89feffe3cb None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:50.908616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c0e089-ca57-4c60-a6b7-ca89feffe3cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:50.939177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e3c0e089-ca57-4c60-a6b7-ca89feffe3cb None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:30:50.941772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e3c0e089-ca57-4c60-a6b7-ca89feffe3cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:30:51.078609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 604/1207] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:30:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:30:51.441973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c06f71e-28cd-4d88-8c2f-86f6a8c05cc9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:51.609817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c06f71e-28cd-4d88-8c2f-86f6a8c05cc9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:51.659885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa806bcf-d5f7-4216-aa56-a3f3340ce8e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:51.661177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa806bcf-d5f7-4216-aa56-a3f3340ce8e5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:51.661177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa806bcf-d5f7-4216-aa56-a3f3340ce8e5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:51.661356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa806bcf-d5f7-4216-aa56-a3f3340ce8e5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:51.686840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa806bcf-d5f7-4216-aa56-a3f3340ce8e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-aa806bcf-d5f7-4216-aa56-a3f3340ce8e5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:30:51.696998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:51.698002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:30:51.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 604/1208] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:30:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:51.705252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c0e089-ca57-4c60-a6b7-ca89feffe3cb None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:51.767118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:51.874178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c0e089-ca57-4c60-a6b7-ca89feffe3cb None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:51.943678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:51.944911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:51.945456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:51.946155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:51.947544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] RBAC: Adding query filter params (name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:51.948084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:51.951089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:51.966443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 605/1209] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:30:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:30:51.975832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa806bcf-d5f7-4216-aa56-a3f3340ce8e5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:52.071150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:52.155881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa806bcf-d5f7-4216-aa56-a3f3340ce8e5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:52.209923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:52.210606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:52.211348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:52.211685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:52.228823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:52.230316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:30:52.249483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:52.251097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:30:52.252071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:30:52.252071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:52.252071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:30:52.252071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 605/1210] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:30:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:52.257307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:52.442923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:52.446992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ea116-9a56-4f8c-a1a2-4fcef696be50 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:52.504878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:52.505634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:52.505915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:52.506183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:52.506934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:52.507273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:52.509760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:52.534569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 606/1211] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:30:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:30:52.540428 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:52.755235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465feaa6-e357-4e58-970c-9c61955a1a99 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:52.758477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:52.846266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:52.847404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:52.849160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:52.849721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] PATH_INFO: `/v3/users/placement` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:52.864774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:52.867446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:30:52.882308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:52.883575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:30:52.884576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 606/1212] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:30:52 2022] GET /identity/v3/users/placement => generated 87 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:30:52.892330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:53.088161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e840d8e6-e0d9-4f44-9fd8-852d40eee671 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:53.140888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:53.141909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:53.143036 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:53.143486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:53.144798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:53.145229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:53.149048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:53.176542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 607/1213] 10.222.0.77 () {56 vars in 1143 bytes} [Mon Aug 29 22:30:52 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:30:53.178595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:53.353619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80239ea1-9a4c-4a04-ad71-a3f870fdc4ce None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:53.420932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2951c577-6d5c-4ecb-9199-bfaf3003639f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:53.421635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2951c577-6d5c-4ecb-9199-bfaf3003639f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:53.421937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2951c577-6d5c-4ecb-9199-bfaf3003639f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:53.422220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2951c577-6d5c-4ecb-9199-bfaf3003639f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:53.467065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:53.538181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2951c577-6d5c-4ecb-9199-bfaf3003639f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:53.540126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2951c577-6d5c-4ecb-9199-bfaf3003639f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:53.711015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 607/1214] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:30:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:30:53.719330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:53.755811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:53.915502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4b5c19-106f-4783-bd37-9eb895e8b540 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:53.985872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:53.986877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:53.987381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:53.987706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:54.008364 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:54.010506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:30:54.036998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:54.038171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:30:54.039155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:30:54.039155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:54.039155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:30:54.043805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 608/1215] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:30:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:54.047093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2951c577-6d5c-4ecb-9199-bfaf3003639f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:54.169721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:54.233042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2951c577-6d5c-4ecb-9199-bfaf3003639f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:54.301867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:54.302537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:54.302955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:54.303231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:54.304164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:54.304542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:54.307004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:54.325621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 608/1216] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:30:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:30:54.337021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:54.450604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:54.492870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d169136-6b4f-4a44-8148-dec7a0e24b8f None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:54.548445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:54.549969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:54.549969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:54.550627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:54.559714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:54.561452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:30:54.576244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:54.577623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:30:54.578711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 609/1217] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:30:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:54.588621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:54.744098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25cfa5d-036a-4c68-ab4a-47e5b0d76656 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:54.778897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:54.806539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac20701-0d4c-4324-b9c3-8f7ecc7cfe33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:54.807658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac20701-0d4c-4324-b9c3-8f7ecc7cfe33 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:54.808145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac20701-0d4c-4324-b9c3-8f7ecc7cfe33 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:54.808642 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac20701-0d4c-4324-b9c3-8f7ecc7cfe33 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:54.810035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac20701-0d4c-4324-b9c3-8f7ecc7cfe33 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:54.810520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac20701-0d4c-4324-b9c3-8f7ecc7cfe33 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:54.813380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac20701-0d4c-4324-b9c3-8f7ecc7cfe33 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:54.829856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 609/1218] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:30:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:30:54.836649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:54.994911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72568ad-de0f-430d-91a5-cded73445b25 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:55.029656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:55.038157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa51c8a-4e32-42bc-83b4-ff14a77983c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:55.038972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa51c8a-4e32-42bc-83b4-ff14a77983c5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:55.045271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa51c8a-4e32-42bc-83b4-ff14a77983c5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:55.045271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa51c8a-4e32-42bc-83b4-ff14a77983c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:55.045271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa51c8a-4e32-42bc-83b4-ff14a77983c5 None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=b6f80303c62847ca9a506d58d9b6f396) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:55.045271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa51c8a-4e32-42bc-83b4-ff14a77983c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:55.045271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa51c8a-4e32-42bc-83b4-ff14a77983c5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:55.056726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 610/1219] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:30:54 2022] GET /identity/v3/role_assignments?user.id=b6f80303c62847ca9a506d58d9b6f396&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:30:57.168390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566d645-0461-4e5b-a324-23ba7cfd6f3e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:57.168390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566d645-0461-4e5b-a324-23ba7cfd6f3e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:57.168390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d566d645-0461-4e5b-a324-23ba7cfd6f3e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:57.169351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 610/1220] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:30:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:30:57.175493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c34f8-006c-4c14-9350-3e9f210d2dbe None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:57.178652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c34f8-006c-4c14-9350-3e9f210d2dbe None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:57.178652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c34f8-006c-4c14-9350-3e9f210d2dbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:57.206767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c74c34f8-006c-4c14-9350-3e9f210d2dbe None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:30:57.208118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c74c34f8-006c-4c14-9350-3e9f210d2dbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:30:57.308119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 611/1221] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:30:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:30:57.594142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d566d645-0461-4e5b-a324-23ba7cfd6f3e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:57.774531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d566d645-0461-4e5b-a324-23ba7cfd6f3e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:57.839770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986ea776-4346-4ec3-a9bf-3da857607af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:57.841011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ea776-4346-4ec3-a9bf-3da857607af8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:57.841476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ea776-4346-4ec3-a9bf-3da857607af8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:57.841922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ea776-4346-4ec3-a9bf-3da857607af8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:57.866043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986ea776-4346-4ec3-a9bf-3da857607af8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-986ea776-4346-4ec3-a9bf-3da857607af8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:30:57.874697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:57.876007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:30:57.879341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 611/1222] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:30:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:30:57.884102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74c34f8-006c-4c14-9350-3e9f210d2dbe None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:58.029675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74c34f8-006c-4c14-9350-3e9f210d2dbe None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:58.077900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:58.098686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:58.099768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:58.100256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:58.100768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:58.102477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] RBAC: Adding query filter params (name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:58.103051 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:58.105981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:58.122893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 612/1223] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:30:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:30:58.129541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986ea776-4346-4ec3-a9bf-3da857607af8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:58.193097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:58.283995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986ea776-4346-4ec3-a9bf-3da857607af8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:58.332300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:58.333035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:58.333355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:58.334358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:58.350417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:58.353984 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:30:58.380298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:58.384141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:30:58.386186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:30:58.386186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:58.386186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:30:58.386186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 612/1224] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:30:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:58.388325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:58.533949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa2b221-06b0-4898-9cb1-244ae67c70c5 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:58.579853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:58.580473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:58.580850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:58.581049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:58.581823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:58.582135 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:58.583723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:58.597722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 613/1225] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:30:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:30:58.606943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:58.768433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:58.779192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815d1bd5-6c21-4987-985b-3cd111064496 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:58.841835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:58.843072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:58.843421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:58.843644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:58.854335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:58.858303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:30:58.876926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:58.878449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:30:58.880141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 613/1226] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:30:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:58.886240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:58.948211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:59.050132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab59d1b-4351-4d80-a4c6-65ee0b32de3b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:59.101438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:59.102566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:59.103076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:59.103528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:59.105032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:59.105617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:59.109599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:59.145108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 614/1227] 10.222.0.77 () {56 vars in 1143 bytes} [Mon Aug 29 22:30:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:30:59.149091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:59.332706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26dea5-9f22-4f33-9a28-171e9bee08e8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:59.393527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:59.394509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:59.394867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:59.395329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:59.409117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:59.410925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:30:59.428522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:59.430919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:30:59.432091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:30:59.432091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:30:59.432091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:30:59.432091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 614/1228] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:30:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:59.438223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:59.574307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d88b24d-b635-4370-a64e-894378f2bf0b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:59.578015 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:59.623921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:59.624589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:59.624893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:59.625229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:59.625888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:30:59.626189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:59.627607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:59.642070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 615/1229] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:30:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:30:59.651700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:30:59.668894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:30:59.822029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7173583-4e00-48c7-8bae-89b5af10f5d2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:30:59.874495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:30:59.875862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:30:59.876271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:30:59.876714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:30:59.892846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:30:59.895827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:30:59.916188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:30:59.918281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:30:59.919985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 615/1230] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:30:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:30:59.924066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:00.073532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77769e12-b4df-4789-b579-1203d2f81051 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:00.133106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6582fb3-bd33-41b3-8eb1-bc0967a6ffb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:00.133106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6582fb3-bd33-41b3-8eb1-bc0967a6ffb9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:00.133106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6582fb3-bd33-41b3-8eb1-bc0967a6ffb9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:00.134618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6582fb3-bd33-41b3-8eb1-bc0967a6ffb9 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:00.134618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6582fb3-bd33-41b3-8eb1-bc0967a6ffb9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:00.134618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6582fb3-bd33-41b3-8eb1-bc0967a6ffb9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:00.135982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6582fb3-bd33-41b3-8eb1-bc0967a6ffb9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:00.153401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 616/1231] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:30:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:00.164322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:00.258956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:00.314302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd07fdb3-b7c8-4957-9215-d195f6ca8d89 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:00.375080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49daee1b-a4c9-43ce-a7c3-5554f6579470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:00.376054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49daee1b-a4c9-43ce-a7c3-5554f6579470 None admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:00.376235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49daee1b-a4c9-43ce-a7c3-5554f6579470 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:00.376463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49daee1b-a4c9-43ce-a7c3-5554f6579470 None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/b6f80303c62847ca9a506d58d9b6f396/roles/6c40acc5170649d8ad758d42c85b91e0` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:00.398367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49daee1b-a4c9-43ce-a7c3-5554f6579470 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=b6f80303c62847ca9a506d58d9b6f396, role_id=6c40acc5170649d8ad758d42c85b91e0)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:00.400662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49daee1b-a4c9-43ce-a7c3-5554f6579470 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:00.421452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 616/1232] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:31:00 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/b6f80303c62847ca9a506d58d9b6f396/roles/6c40acc5170649d8ad758d42c85b91e0 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:31:02.575026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172cffe-4176-4bf3-9697-92366f180a8c None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:02.575875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172cffe-4176-4bf3-9697-92366f180a8c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:02.575875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172cffe-4176-4bf3-9697-92366f180a8c None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:02.576952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 617/1233] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:31:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:31:02.582655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ab1242-df67-4e45-a45b-06ba9e1aba63 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:02.583484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ab1242-df67-4e45-a45b-06ba9e1aba63 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:02.583484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ab1242-df67-4e45-a45b-06ba9e1aba63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:02.608139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-57ab1242-df67-4e45-a45b-06ba9e1aba63 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:31:02.609824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-57ab1242-df67-4e45-a45b-06ba9e1aba63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:31:02.732905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 617/1234] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:31:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:31:03.033890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f172cffe-4176-4bf3-9697-92366f180a8c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:03.071034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:03.210148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f172cffe-4176-4bf3-9697-92366f180a8c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:03.255290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387292cd-fc92-442f-bf15-34b96e47cd6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:03.255939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387292cd-fc92-442f-bf15-34b96e47cd6e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:03.256317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387292cd-fc92-442f-bf15-34b96e47cd6e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:03.256810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387292cd-fc92-442f-bf15-34b96e47cd6e None admin] PATH_INFO: `/v3/roles/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:03.271280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387292cd-fc92-442f-bf15-34b96e47cd6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-387292cd-fc92-442f-bf15-34b96e47cd6e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:31:03.276392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:03.279437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:03.280401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 618/1235] 10.222.0.77 () {56 vars in 1102 bytes} [Mon Aug 29 22:31:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:03.285859 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ab1242-df67-4e45-a45b-06ba9e1aba63 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:03.358333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:03.435347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ab1242-df67-4e45-a45b-06ba9e1aba63 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:03.488096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:03.488801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:03.489183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:03.489461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:03.490148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] RBAC: Adding query filter params (name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:03.490459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:03.491957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:03.501673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 618/1236] 10.222.0.77 () {56 vars in 1103 bytes} [Mon Aug 29 22:31:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:03.508187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387292cd-fc92-442f-bf15-34b96e47cd6e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:03.624804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:03.644950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387292cd-fc92-442f-bf15-34b96e47cd6e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:03.692288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:03.692967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:03.693284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:03.693560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:03.706116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:03.707782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:31:03.726549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:03.727718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:31:03.728812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:31:03.728812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:03.728812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:03.728812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 619/1237] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:03.738319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:03.884329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf8cf30-d4e2-4c63-ad0b-f4a25fb927fa None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:03.898864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:03.934595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:03.935683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:03.936193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:03.936752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:03.938007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:03.938556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:03.941954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:03.964974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 619/1238] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:31:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:03.971172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:04.096576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e1c02d-bbf4-4b66-83c7-a5d4008894ab None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:04.139570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:04.140111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:04.140439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:04.140743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] PATH_INFO: `/v3/users/placement` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:04.144309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:04.151899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:04.153545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:31:04.167419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:04.173439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:04.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 620/1239] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:03 2022] GET /identity/v3/users/placement => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:04.175427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:04.303131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4af51e-abdb-4b80-9c82-d38b58509660 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:04.352790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:04.353744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:04.353968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:04.354368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:04.355311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:04.355708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:04.357391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:04.382339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 620/1240] 10.222.0.77 () {56 vars in 1143 bytes} [Mon Aug 29 22:31:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:04.394230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:04.535890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f45179-3506-4ece-8fc2-1278947673b0 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:04.591000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32e2b01-ecfc-4184-afa0-108deeeb7893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:04.592196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e2b01-ecfc-4184-afa0-108deeeb7893 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:04.592657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e2b01-ecfc-4184-afa0-108deeeb7893 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:04.593110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e2b01-ecfc-4184-afa0-108deeeb7893 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:04.690091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:04.697694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32e2b01-ecfc-4184-afa0-108deeeb7893 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:04.699326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32e2b01-ecfc-4184-afa0-108deeeb7893 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:04.860245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 621/1241] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:31:04 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:31:04.870046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:04.913201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:05.000530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542dc549-c482-46c0-9aa7-7f6c6a3c41d1 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:05.047364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:05.048027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:05.048443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:05.048599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:05.066953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:05.068440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:31:05.087604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:05.090043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:31:05.091809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:31:05.091809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:05.091809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:05.091809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 621/1242] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:31:05.096625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32e2b01-ecfc-4184-afa0-108deeeb7893 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:05.238344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:05.253797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32e2b01-ecfc-4184-afa0-108deeeb7893 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:05.300337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:05.300965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:05.301281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:05.301555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:05.302241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:05.302627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:05.304157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:05.318542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 622/1243] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:31:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:05.325614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:05.457605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:05.484725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf6ca4a-4d03-4b16-b9ba-d719f2cd4991 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:05.528486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:05.529011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:05.529297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:05.529582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:05.537870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:05.539393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:31:05.553570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:05.555519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:05.557406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 622/1244] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:31:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:05.561764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:05.676535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba787540-2f32-4391-a2aa-811d411d0597 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:05.717019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4158c48-0c42-4ed0-a79c-5485a2b7ea12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:05.718243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4158c48-0c42-4ed0-a79c-5485a2b7ea12 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:05.718243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4158c48-0c42-4ed0-a79c-5485a2b7ea12 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:05.718243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4158c48-0c42-4ed0-a79c-5485a2b7ea12 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:05.718985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4158c48-0c42-4ed0-a79c-5485a2b7ea12 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:05.719297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4158c48-0c42-4ed0-a79c-5485a2b7ea12 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:05.720838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4158c48-0c42-4ed0-a79c-5485a2b7ea12 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:05.734718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 623/1245] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:31:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:05.741370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:05.874290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1794539-2b0b-46b1-a54d-f2e0ca205cc3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:05.911950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152751a8-c838-4847-84a9-3a77c2a1a5f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:05.912577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152751a8-c838-4847-84a9-3a77c2a1a5f7 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:05.912863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152751a8-c838-4847-84a9-3a77c2a1a5f7 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:05.913134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152751a8-c838-4847-84a9-3a77c2a1a5f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:05.914102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152751a8-c838-4847-84a9-3a77c2a1a5f7 None admin] RBAC: Adding query filter params (role.id=6c40acc5170649d8ad758d42c85b91e0, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=b6f80303c62847ca9a506d58d9b6f396) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:05.917791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152751a8-c838-4847-84a9-3a77c2a1a5f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:05.917791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152751a8-c838-4847-84a9-3a77c2a1a5f7 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:05.923598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 623/1246] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:31:05 2022] GET /identity/v3/role_assignments?user.id=b6f80303c62847ca9a506d58d9b6f396&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=6c40acc5170649d8ad758d42c85b91e0 => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:31:08.282390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b11c7-9aeb-4c48-ae32-c1ab0ab29617 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:08.283027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b11c7-9aeb-4c48-ae32-c1ab0ab29617 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:08.283130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2b11c7-9aeb-4c48-ae32-c1ab0ab29617 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:08.284710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 624/1247] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:31:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:31:08.290918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedd4e06-fc12-464f-84df-7c06b256865e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:08.291447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedd4e06-fc12-464f-84df-7c06b256865e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:08.291447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedd4e06-fc12-464f-84df-7c06b256865e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:08.299019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:08.322133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-aedd4e06-fc12-464f-84df-7c06b256865e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:31:08.323538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-aedd4e06-fc12-464f-84df-7c06b256865e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:31:08.443573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 624/1248] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:31:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:31:08.736152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2b11c7-9aeb-4c48-ae32-c1ab0ab29617 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:08.763026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:08.896450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2b11c7-9aeb-4c48-ae32-c1ab0ab29617 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:08.944201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf40d3b-19f4-45fc-a7c3-8a3055e03bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:08.945053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf40d3b-19f4-45fc-a7c3-8a3055e03bdc None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:08.946128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf40d3b-19f4-45fc-a7c3-8a3055e03bdc None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:08.946651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf40d3b-19f4-45fc-a7c3-8a3055e03bdc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:08.964080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf40d3b-19f4-45fc-a7c3-8a3055e03bdc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-9cf40d3b-19f4-45fc-a7c3-8a3055e03bdc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:31:08.970055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:08.972141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:08.973573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 625/1249] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:31:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:08.983525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedd4e06-fc12-464f-84df-7c06b256865e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:09.121919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedd4e06-fc12-464f-84df-7c06b256865e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:09.161361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:09.167854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:09.168487 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:09.168782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:09.169249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:09.169705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:09.170002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:09.171528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:09.185439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 625/1250] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:31:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:09.194532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf40d3b-19f4-45fc-a7c3-8a3055e03bdc None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:09.296865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:09.330747 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf40d3b-19f4-45fc-a7c3-8a3055e03bdc None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:09.383965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:09.384734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:09.385025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:09.385301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:09.400491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:09.402320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:31:09.421576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:09.428028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:31:09.429239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:31:09.429239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:09.429239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:09.429239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 626/1251] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:31:09.430627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:09.604345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ba6665-6f49-4eca-85fa-266207997bb3 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:09.652834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:09.653386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:09.653666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:09.653936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:09.654718 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:09.655103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:09.656591 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:09.677093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 626/1252] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:31:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:09.684054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:09.838941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eff95-f322-4405-b9c6-62d3e021b8b1 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:09.875900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:09.888299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:09.888990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:09.889280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:09.889555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] PATH_INFO: `/v3/users/placement` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:09.903091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:09.904514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:31:09.920426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:09.921830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:09.923400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 627/1253] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:09 2022] GET /identity/v3/users/placement => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:09.927879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:10.015918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:10.068986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63569f4d-eeb6-4a27-a2de-77920741a5c2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:10.117153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:10.118149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:10.118612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:10.119176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:10.120297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:10.120786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:10.122571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:10.150744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 627/1254] 10.222.0.77 () {56 vars in 1143 bytes} [Mon Aug 29 22:31:09 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:10.157129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:10.331634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f92dc5-39ac-409a-bd67-f0cca9c4ff55 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:10.406475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0db806f-b8b6-4f64-9075-c784730fd58b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:10.407211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db806f-b8b6-4f64-9075-c784730fd58b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:10.407486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db806f-b8b6-4f64-9075-c784730fd58b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:10.407876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db806f-b8b6-4f64-9075-c784730fd58b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:10.505974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:10.527463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0db806f-b8b6-4f64-9075-c784730fd58b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:10.529650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0db806f-b8b6-4f64-9075-c784730fd58b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:10.687813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 628/1255] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:31:10 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 531 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 29 22:31:10.695606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:10.840876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f4f6f1-3704-4503-b198-5f744116a095 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:10.864026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:10.903013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:10.903992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:10.904370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:10.904811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:10.923976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:10.926078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:31:10.954117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:10.956562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:31:10.959560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:31:10.959560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:10.959560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:10.959560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 628/1256] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:10.963816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0db806f-b8b6-4f64-9075-c784730fd58b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:11.121575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:11.141783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0db806f-b8b6-4f64-9075-c784730fd58b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:11.208764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:11.209528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:11.209868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:11.210273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:11.211019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:11.211400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:11.213956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:11.235874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 629/1257] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:31:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:11.237198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:11.407580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6acdf-50a8-49e5-9617-00de1af7d9a0 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:11.466559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:11.467268 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:11.467668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:11.468037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:11.482608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:11.482608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:31:11.497013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:11.498155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:11.499244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 629/1258] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:31:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:11.505570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:11.668983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d017b42-9293-4a54-b131-fce48b983b80 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:11.722963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8af0d96-ddc4-4158-890f-f8c8560f2603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:11.723618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8af0d96-ddc4-4158-890f-f8c8560f2603 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:11.723894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8af0d96-ddc4-4158-890f-f8c8560f2603 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:11.724165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8af0d96-ddc4-4158-890f-f8c8560f2603 None admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:11.725014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8af0d96-ddc4-4158-890f-f8c8560f2603 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:11.725503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8af0d96-ddc4-4158-890f-f8c8560f2603 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:11.727173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8af0d96-ddc4-4158-890f-f8c8560f2603 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:11.757881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 630/1259] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:31:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:31:11.759529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:11.813341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:11.930102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adeaf01-0d22-4ca5-ab12-6d7b7ddd1e39 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:11.973752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2f59f8-bc30-4bcb-aac2-6ff652efc2f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:11.974444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f59f8-bc30-4bcb-aac2-6ff652efc2f8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:11.974807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f59f8-bc30-4bcb-aac2-6ff652efc2f8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:11.975680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f59f8-bc30-4bcb-aac2-6ff652efc2f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:11.976621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2f59f8-bc30-4bcb-aac2-6ff652efc2f8 None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=b6f80303c62847ca9a506d58d9b6f396) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:11.977062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2f59f8-bc30-4bcb-aac2-6ff652efc2f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:11.978787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2f59f8-bc30-4bcb-aac2-6ff652efc2f8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:11.986071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 630/1260] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:31:11 2022] GET /identity/v3/role_assignments?user.id=b6f80303c62847ca9a506d58d9b6f396&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=38171c89968943bd92d9e9758db65615 => generated 259 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:14.141519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ea3661-ef47-46b2-afb7-8539768afdcb None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:14.142275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ea3661-ef47-46b2-afb7-8539768afdcb None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:14.142275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ea3661-ef47-46b2-afb7-8539768afdcb None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:14.143332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 631/1261] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:31:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:31:14.150457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c40b943-1fab-4158-8e37-3be4549e04ea None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:14.151047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c40b943-1fab-4158-8e37-3be4549e04ea None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:14.151388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c40b943-1fab-4158-8e37-3be4549e04ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:14.173541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9c40b943-1fab-4158-8e37-3be4549e04ea None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:31:14.174772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9c40b943-1fab-4158-8e37-3be4549e04ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:31:14.266022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 631/1262] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:31:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:31:14.541667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ea3661-ef47-46b2-afb7-8539768afdcb None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:14.557671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:14.661904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ea3661-ef47-46b2-afb7-8539768afdcb None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:14.701512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726f41a5-af56-4fd8-b972-918766dbfa46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:14.702159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726f41a5-af56-4fd8-b972-918766dbfa46 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:14.702439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726f41a5-af56-4fd8-b972-918766dbfa46 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:14.702707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726f41a5-af56-4fd8-b972-918766dbfa46 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:14.714721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726f41a5-af56-4fd8-b972-918766dbfa46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-726f41a5-af56-4fd8-b972-918766dbfa46 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:31:14.719668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:14.720874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:14.722063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 632/1263] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:31:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:14.727614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c40b943-1fab-4158-8e37-3be4549e04ea None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:14.755992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:14.857610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c40b943-1fab-4158-8e37-3be4549e04ea None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:14.900042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:14.900653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:14.900928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:14.901196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:14.901945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:14.902383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:14.903935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:14.913698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 632/1264] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:31:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:14.919912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726f41a5-af56-4fd8-b972-918766dbfa46 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:14.995945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:15.035572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726f41a5-af56-4fd8-b972-918766dbfa46 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:15.075166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:15.075709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:15.075990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:15.076304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:15.088205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:15.089806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:31:15.108654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:15.110449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:31:15.112278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:31:15.112278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:15.112278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:15.112278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 633/1265] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:15.119831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:15.211539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:15.233138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68957985-71ee-466b-ad88-2d4011818fd8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:15.270813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:15.271432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:15.271712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:15.271978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:15.272622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:15.272924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:15.274495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:15.290953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 633/1266] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:31:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:15.297436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:15.439586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e1b3b3-95e8-49e8-a0f7-dc60a5a532d9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:15.460770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:15.476094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:15.476734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:15.477029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:15.477430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] PATH_INFO: `/v3/users/placement` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:15.485600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:15.487267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:31:15.500807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:15.502666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:15.504553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 634/1267] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:15 2022] GET /identity/v3/users/placement => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:15.508778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:15.622133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed0257d-b7e0-443b-8a24-0ef163f19993 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:15.661895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:15.662511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:15.662825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:15.663098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:15.663841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:15.664175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:15.665695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:15.683237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:15.693660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 634/1268] 10.222.0.77 () {56 vars in 1143 bytes} [Mon Aug 29 22:31:15 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:31:15.700646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:15.834529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb65433-cbb3-40d5-9490-081d67bfbcae None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:15.872516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:15.873154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:15.873433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:15.873704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:15.885432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:15.886920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:31:15.901979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:15.903912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:31:15.906016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:31:15.906016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:15.906016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:15.906016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 635/1269] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:15.912392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:16.027052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1b89ba-5f24-43d2-bbcb-a3a8b3764ef5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:16.070440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:16.071172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:16.071448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:16.071714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:16.072373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:16.072667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:16.074056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:16.087862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 635/1270] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:31:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:16.094275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:16.141373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:16.239051 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20fdbcf-5c89-4cd7-bd99-969918ecb672 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:16.276500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:16.277613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:16.278062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:16.278544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:16.288638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:16.290109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:31:16.302118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:16.304253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:16.305468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 636/1271] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:31:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:16.312342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:16.390195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:16.446721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b78800e-8ff5-4539-84be-ab40bf9ddf91 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:16.488948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44931c18-4bda-41e0-97d2-5fe8cbc53180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:16.489587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44931c18-4bda-41e0-97d2-5fe8cbc53180 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:16.489870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44931c18-4bda-41e0-97d2-5fe8cbc53180 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:16.490144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44931c18-4bda-41e0-97d2-5fe8cbc53180 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:16.490906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44931c18-4bda-41e0-97d2-5fe8cbc53180 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:16.491213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44931c18-4bda-41e0-97d2-5fe8cbc53180 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:16.493167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44931c18-4bda-41e0-97d2-5fe8cbc53180 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:16.509331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 636/1272] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:31:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:16.518676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:16.630834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:16.641823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173a44e1-add1-40f9-96c5-20ade2654927 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:16.690466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc65f7-8bc8-4b4d-9c75-e6016536b776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:16.691973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc65f7-8bc8-4b4d-9c75-e6016536b776 None admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:16.692262 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc65f7-8bc8-4b4d-9c75-e6016536b776 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:16.692544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc65f7-8bc8-4b4d-9c75-e6016536b776 None admin] PATH_INFO: `/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/b6f80303c62847ca9a506d58d9b6f396/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:16.694526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfc65f7-8bc8-4b4d-9c75-e6016536b776 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d28d10d57ea42408a6ced32a39139c9, user_id=b6f80303c62847ca9a506d58d9b6f396, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:16.696391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfc65f7-8bc8-4b4d-9c75-e6016536b776 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:16.713287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 637/1273] 10.222.0.77 () {58 vars in 1439 bytes} [Mon Aug 29 22:31:16 2022] PUT /identity/v3/projects/3d28d10d57ea42408a6ced32a39139c9/users/b6f80303c62847ca9a506d58d9b6f396/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:31:18.670136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b93637-461b-4dc2-ba12-8444f3c5a04d None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:18.670725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b93637-461b-4dc2-ba12-8444f3c5a04d None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:18.670725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b93637-461b-4dc2-ba12-8444f3c5a04d None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:18.672023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 637/1274] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:31:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:31:18.677845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a2f958-1b0e-49ba-9a7b-4804db97d5c6 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:18.678528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a2f958-1b0e-49ba-9a7b-4804db97d5c6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:18.678528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a2f958-1b0e-49ba-9a7b-4804db97d5c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:18.699055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-82a2f958-1b0e-49ba-9a7b-4804db97d5c6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:31:18.700268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-82a2f958-1b0e-49ba-9a7b-4804db97d5c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:31:18.774054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 638/1275] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:31:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 29 22:31:19.046207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b93637-461b-4dc2-ba12-8444f3c5a04d None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:19.189069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b93637-461b-4dc2-ba12-8444f3c5a04d None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:19.208667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:19.228074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afda91a-a0b4-4e12-85a3-c0c748235ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:19.228758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afda91a-a0b4-4e12-85a3-c0c748235ae6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:19.229082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afda91a-a0b4-4e12-85a3-c0c748235ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:19.229402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afda91a-a0b4-4e12-85a3-c0c748235ae6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:19.241140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afda91a-a0b4-4e12-85a3-c0c748235ae6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-8afda91a-a0b4-4e12-85a3-c0c748235ae6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 29 22:31:19.246092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:19.248072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:19.250052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 638/1276] 10.222.0.77 () {56 vars in 1096 bytes} [Mon Aug 29 22:31:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:19.255292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a2f958-1b0e-49ba-9a7b-4804db97d5c6 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:19.374042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:19.399554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a2f958-1b0e-49ba-9a7b-4804db97d5c6 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:19.440983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:19.441587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:19.441862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:19.442125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:19.442810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:19.443115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:19.444488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:19.453287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 639/1277] 10.222.0.77 () {56 vars in 1099 bytes} [Mon Aug 29 22:31:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:19.460679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afda91a-a0b4-4e12-85a3-c0c748235ae6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:19.595900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afda91a-a0b4-4e12-85a3-c0c748235ae6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:19.641970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:19.642491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:19.642785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:19.643609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:19.655236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:19.656698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:31:19.672492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:19.674320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:31:19.675581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:31:19.675581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:19.675581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:19.675581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 639/1278] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:19.679845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:19.795079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f7f8fc-e5f6-4806-8efe-b0beb8571ea9 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:19.850481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:19.851543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:19.852037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:19.852507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:19.853681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:19.854205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:19.856699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:19.872014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 640/1279] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:31:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:31:19.879289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:19.918437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:20.010103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5548247f-b4f9-4d89-9778-6150ef63099f None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:20.062272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:20.063008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:20.063335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:20.063653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:20.075200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:20.076810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 29 22:31:20.089156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:20.091169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:20.093230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 640/1280] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:19 2022] GET /identity/v3/users/placement => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:31:20.096730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:20.101203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:20.229534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f5c7a3-c085-4706-b66c-c2d31bd7602e None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:20.267085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:20.267701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:20.267990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:20.268337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:20.269000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:20.269308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:20.270802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:20.290178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 641/1281] 10.222.0.77 () {56 vars in 1143 bytes} [Mon Aug 29 22:31:20 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:20.296315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:20.431909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:20.445104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edba6998-f455-4142-9186-f900fc4f9db2 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:20.487333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b63418f-e52c-43f1-9905-79ad3d299b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:20.487844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b63418f-e52c-43f1-9905-79ad3d299b38 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:20.488134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b63418f-e52c-43f1-9905-79ad3d299b38 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:20.488406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b63418f-e52c-43f1-9905-79ad3d299b38 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:20.576255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b63418f-e52c-43f1-9905-79ad3d299b38 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:20.577708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b63418f-e52c-43f1-9905-79ad3d299b38 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:20.691447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:20.710162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 641/1282] 10.222.0.77 () {58 vars in 1282 bytes} [Mon Aug 29 22:31:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 29 22:31:20.716135 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:20.788286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:20.843425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e742778e-27c7-4ff3-a0cc-ad4500c654fe None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:20.879072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:20.879689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:20.880057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:20.880326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:20.890630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:20.892101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 29 22:31:20.907743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:20.910482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 29 22:31:20.912127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 29 22:31:20.912127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 29 22:31:20.912127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:20.912127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 642/1283] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:31:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 29 22:31:20.915816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b63418f-e52c-43f1-9905-79ad3d299b38 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:21.035136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b63418f-e52c-43f1-9905-79ad3d299b38 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:21.075050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:21.075544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:21.075825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:21.076095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:21.076818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:21.077122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:21.078499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:21.090262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 642/1284] 10.222.0.77 () {56 vars in 1109 bytes} [Mon Aug 29 22:31:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:21.097839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:21.217194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5015353c-5f3b-45ab-a37a-e7ff5ec0449b None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:21.240537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:21.274056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:21.275128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:21.275608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:21.276075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] PATH_INFO: `/v3/projects/service` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:21.288208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:21.290881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 29 22:31:21.311860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:21.312965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: ERROR keystone.server.flask.application Aug 29 22:31:21.314039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 643/1285] 10.222.0.77 () {56 vars in 1111 bytes} [Mon Aug 29 22:31:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:21.320056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:21.376444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:21.443589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1631c7-257f-498c-a4cd-0fa56d42b1a5 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:21.493760 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafbc174-c63f-4df1-be5a-625fdeaf7166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:21.496552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafbc174-c63f-4df1-be5a-625fdeaf7166 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:21.497584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafbc174-c63f-4df1-be5a-625fdeaf7166 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:21.498397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafbc174-c63f-4df1-be5a-625fdeaf7166 None admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:21.500398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafbc174-c63f-4df1-be5a-625fdeaf7166 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:21.501735 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafbc174-c63f-4df1-be5a-625fdeaf7166 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:21.505341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafbc174-c63f-4df1-be5a-625fdeaf7166 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:21.524430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 643/1286] 10.222.0.77 () {56 vars in 1148 bytes} [Mon Aug 29 22:31:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:21.530958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:21.651166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629c9c49-fba2-4396-8539-080b19fbc928 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:21.689676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec92485-26f1-4c58-8a5b-1f8328ebb51e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:21.690281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec92485-26f1-4c58-8a5b-1f8328ebb51e None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:21.690566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec92485-26f1-4c58-8a5b-1f8328ebb51e None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:21.690872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec92485-26f1-4c58-8a5b-1f8328ebb51e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:21.691606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec92485-26f1-4c58-8a5b-1f8328ebb51e None admin] RBAC: Adding query filter params (role.id=38171c89968943bd92d9e9758db65615, scope.project.id=3d28d10d57ea42408a6ced32a39139c9, user.id=b6f80303c62847ca9a506d58d9b6f396) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:21.691920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec92485-26f1-4c58-8a5b-1f8328ebb51e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:21.693361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec92485-26f1-4c58-8a5b-1f8328ebb51e None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:21.699617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 644/1287] 10.222.0.77 () {56 vars in 1374 bytes} [Mon Aug 29 22:31:21 2022] GET /identity/v3/role_assignments?user.id=b6f80303c62847ca9a506d58d9b6f396&scope.project.id=3d28d10d57ea42408a6ced32a39139c9&role.id=38171c89968943bd92d9e9758db65615 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:31:23.791479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b779d-a9c0-43c0-bd39-2d49ca8d3c7e None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:23.792159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b779d-a9c0-43c0-bd39-2d49ca8d3c7e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:23.792159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b779d-a9c0-43c0-bd39-2d49ca8d3c7e None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:23.793512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 644/1288] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:31:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:31:23.802308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00218ec-22d4-4973-8858-4deeb56d19c1 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:23.802897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00218ec-22d4-4973-8858-4deeb56d19c1 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:23.803308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00218ec-22d4-4973-8858-4deeb56d19c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:23.833311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a00218ec-22d4-4973-8858-4deeb56d19c1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:31:23.835378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a00218ec-22d4-4973-8858-4deeb56d19c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:31:23.872200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:23.919216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 645/1289] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:31:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:31:24.187563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b779d-a9c0-43c0-bd39-2d49ca8d3c7e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:24.296678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b779d-a9c0-43c0-bd39-2d49ca8d3c7e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:24.309581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:24.340945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:24.341593 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:24.341978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:24.342254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] PATH_INFO: `/v3/services/placement` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:24.342882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:24.344834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:24.361971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:24.363994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:24.365360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 645/1290] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:31:24 2022] GET /identity/v3/services/placement => generated 90 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:24.373083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00218ec-22d4-4973-8858-4deeb56d19c1 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:24.518831 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00218ec-22d4-4973-8858-4deeb56d19c1 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:24.569109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2f2047-6bae-4673-af23-a38bb5ef1975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:24.569730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f2047-6bae-4673-af23-a38bb5ef1975 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:24.570017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f2047-6bae-4673-af23-a38bb5ef1975 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:24.570287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f2047-6bae-4673-af23-a38bb5ef1975 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:24.570970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2f2047-6bae-4673-af23-a38bb5ef1975 None admin] RBAC: Adding query filter params (name=placement) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:24.571274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2f2047-6bae-4673-af23-a38bb5ef1975 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:24.572659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2f2047-6bae-4673-af23-a38bb5ef1975 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:24.578380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 646/1291] 10.222.0.77 () {56 vars in 1116 bytes} [Mon Aug 29 22:31:24 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:24.585112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:24.702307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dc2df2-07a9-45d7-ae37-28d4ab8286dd None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:24.752377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1173e83-86a4-4ed2-ae15-e79a4b228fbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:24.752976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1173e83-86a4-4ed2-ae15-e79a4b228fbf None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:24.753253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1173e83-86a4-4ed2-ae15-e79a4b228fbf None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:24.753523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1173e83-86a4-4ed2-ae15-e79a4b228fbf None admin] PATH_INFO: `/v3/services` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:24.754291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1173e83-86a4-4ed2-ae15-e79a4b228fbf None admin] RBAC: Adding query filter params (type=placement) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:24.754596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1173e83-86a4-4ed2-ae15-e79a4b228fbf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:24.756011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1173e83-86a4-4ed2-ae15-e79a4b228fbf None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:24.760689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 646/1292] 10.222.0.77 () {56 vars in 1116 bytes} [Mon Aug 29 22:31:24 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:31:26.869535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d5f73-f226-4d68-9eca-88554327a5a8 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:26.870099 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d5f73-f226-4d68-9eca-88554327a5a8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:26.870099 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d5f73-f226-4d68-9eca-88554327a5a8 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:26.872480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 647/1293] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:31:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:31:26.881823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec04f23a-8b39-4b1b-a893-a5df4f47cd98 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:26.882356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec04f23a-8b39-4b1b-a893-a5df4f47cd98 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:26.882701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec04f23a-8b39-4b1b-a893-a5df4f47cd98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:26.910841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-ec04f23a-8b39-4b1b-a893-a5df4f47cd98 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:31:26.912952 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ec04f23a-8b39-4b1b-a893-a5df4f47cd98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:31:26.951985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:26.993361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 647/1294] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:31:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:31:27.273114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61d5f73-f226-4d68-9eca-88554327a5a8 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:27.306600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:27.417282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61d5f73-f226-4d68-9eca-88554327a5a8 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:27.456881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1913d52f-96ee-481f-898b-05f4d7435e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:27.457487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913d52f-96ee-481f-898b-05f4d7435e59 None admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:27.457765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913d52f-96ee-481f-898b-05f4d7435e59 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:27.458034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913d52f-96ee-481f-898b-05f4d7435e59 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:27.458705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1913d52f-96ee-481f-898b-05f4d7435e59 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:27.460179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1913d52f-96ee-481f-898b-05f4d7435e59 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:27.477101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 648/1295] 10.222.0.77 () {60 vars in 1141 bytes} [Mon Aug 29 22:31:27 2022] POST /identity/v3/services => generated 246 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:31:29.449143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7647b3e-9853-4ddb-84f5-433a3eaa67af None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:29.450134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7647b3e-9853-4ddb-84f5-433a3eaa67af None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:29.450134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7647b3e-9853-4ddb-84f5-433a3eaa67af None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:29.451828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 648/1296] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:31:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:31:29.469382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8de603-535a-4634-8bd4-cb02f26bfe11 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:29.470040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8de603-535a-4634-8bd4-cb02f26bfe11 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:29.470396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8de603-535a-4634-8bd4-cb02f26bfe11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:29.501752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7f8de603-535a-4634-8bd4-cb02f26bfe11 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:31:29.503829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7f8de603-535a-4634-8bd4-cb02f26bfe11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:31:29.610753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 649/1297] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:31:29 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:31:29.884857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7647b3e-9853-4ddb-84f5-433a3eaa67af None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:30.004807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7647b3e-9853-4ddb-84f5-433a3eaa67af None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:30.042019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:30.042641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:30.042948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:30.043220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] PATH_INFO: `/v3/services/placement` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:30.043813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:30.045265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:30.054425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:30.056243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:30.057444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 649/1298] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:31:29 2022] GET /identity/v3/services/placement => generated 90 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:30.062200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8de603-535a-4634-8bd4-cb02f26bfe11 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:30.090659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:30.193624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8de603-535a-4634-8bd4-cb02f26bfe11 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:30.233299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:30.233981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:30.234313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:30.234613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:30.235378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] RBAC: Adding query filter params (name=placement) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:30.235845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:30.237698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:30.242963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 650/1299] 10.222.0.77 () {56 vars in 1116 bytes} [Mon Aug 29 22:31:30 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:30.252290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:30.285186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:30.370945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f60d57-e3a6-4725-857e-968053ae3ad8 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:30.420416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51658978-c2d7-4225-a370-edb41cea980f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:30.421212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51658978-c2d7-4225-a370-edb41cea980f None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:30.421680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51658978-c2d7-4225-a370-edb41cea980f None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:30.422150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51658978-c2d7-4225-a370-edb41cea980f None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:30.423497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51658978-c2d7-4225-a370-edb41cea980f None admin] RBAC: Adding query filter params (interface=public, service_id=b836136c7f9e438ca9741f8148376b06, region_id=RegionOne) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:30.424034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51658978-c2d7-4225-a370-edb41cea980f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:30.426150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51658978-c2d7-4225-a370-edb41cea980f None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:30.436774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 650/1300] 10.222.0.77 () {56 vars in 1251 bytes} [Mon Aug 29 22:31:30 2022] GET /identity/v3/endpoints?service_id=b836136c7f9e438ca9741f8148376b06&interface=public®ion_id=RegionOne => generated 194 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:30.444024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:30.568453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:30.597580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7afa439-67b5-4be0-93b2-2e9867711c37 None admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:30.655862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111fb1fd-2bfa-414c-8c96-548e61b05426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:30.656356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111fb1fd-2bfa-414c-8c96-548e61b05426 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:30.656637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111fb1fd-2bfa-414c-8c96-548e61b05426 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:30.656911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111fb1fd-2bfa-414c-8c96-548e61b05426 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:30.657523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111fb1fd-2bfa-414c-8c96-548e61b05426 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:30.658948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111fb1fd-2bfa-414c-8c96-548e61b05426 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:30.671746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 651/1301] 10.222.0.77 () {56 vars in 1087 bytes} [Mon Aug 29 22:31:30 2022] GET /identity/v3/services => generated 2191 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:31:32.687555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f595ab-cb80-4e20-ba80-4dd58d19e58b None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:32.688123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f595ab-cb80-4e20-ba80-4dd58d19e58b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:32.688123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f595ab-cb80-4e20-ba80-4dd58d19e58b None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:32.689429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 651/1302] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:31:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:31:32.694644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8fa68-fdf4-4bd8-89b6-d90aabdbf00b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:32.695268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8fa68-fdf4-4bd8-89b6-d90aabdbf00b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:32.695268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8fa68-fdf4-4bd8-89b6-d90aabdbf00b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:32.714563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e5d8fa68-fdf4-4bd8-89b6-d90aabdbf00b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:31:32.715861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e5d8fa68-fdf4-4bd8-89b6-d90aabdbf00b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:31:32.810189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 652/1303] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:31:32 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 29 22:31:33.089335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f595ab-cb80-4e20-ba80-4dd58d19e58b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:33.204670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:33.213898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f595ab-cb80-4e20-ba80-4dd58d19e58b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:33.257653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:33.258294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:33.258569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:33.258876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] PATH_INFO: `/v3/services/placement` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:33.259466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:33.261773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 29 22:31:33.277549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 29 22:31:33.279523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.get_or_create( Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application with Lock( Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self._enter() Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application return self.creator() Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application created_value = creator( Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: ERROR keystone.server.flask.application Aug 29 22:31:33.281181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 652/1304] 10.222.0.77 () {56 vars in 1117 bytes} [Mon Aug 29 22:31:33 2022] GET /identity/v3/services/placement => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 29 22:31:33.285210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8fa68-fdf4-4bd8-89b6-d90aabdbf00b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:33.370272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:33.396669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8fa68-fdf4-4bd8-89b6-d90aabdbf00b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:33.437748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca47bc6-b899-450f-8006-947e91ac0ee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:33.438371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca47bc6-b899-450f-8006-947e91ac0ee6 None admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:33.438911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca47bc6-b899-450f-8006-947e91ac0ee6 None admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:33.439190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca47bc6-b899-450f-8006-947e91ac0ee6 None admin] PATH_INFO: `/v3/services` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:33.439842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca47bc6-b899-450f-8006-947e91ac0ee6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:31:33.440150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca47bc6-b899-450f-8006-947e91ac0ee6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:33.441694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca47bc6-b899-450f-8006-947e91ac0ee6 None admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:33.447762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 653/1305] 10.222.0.77 () {56 vars in 1116 bytes} [Mon Aug 29 22:31:33 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:31:33.454573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:31:33.577092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a884d82-b355-4dde-85bf-4234a8fef6e6 None admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:31:33.626203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f81e5cf-b995-4042-a0bd-a24259fbc079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:31:33.626818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81e5cf-b995-4042-a0bd-a24259fbc079 None admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:31:33.627135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81e5cf-b995-4042-a0bd-a24259fbc079 None admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:31:33.627444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81e5cf-b995-4042-a0bd-a24259fbc079 None admin] PATH_INFO: `/v3/endpoints` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:31:33.628197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f81e5cf-b995-4042-a0bd-a24259fbc079 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:31:33.629613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f81e5cf-b995-4042-a0bd-a24259fbc079 None admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:31:33.652713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 653/1306] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:31:33 2022] POST /identity/v3/endpoints => generated 331 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:31:43.444506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=30 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:31:43.647102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:45.253869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72b556-b379-44d0-83c5-367955cf037f None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:45.255077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72b556-b379-44d0-83c5-367955cf037f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:45.255596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72b556-b379-44d0-83c5-367955cf037f None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:45.258357 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 654/1307] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:32:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:32:45.267646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4a3001-0d44-4a58-9ecf-2c9c27b32d62 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:45.268410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4a3001-0d44-4a58-9ecf-2c9c27b32d62 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:45.268410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4a3001-0d44-4a58-9ecf-2c9c27b32d62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:45.377252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-0f4a3001-0d44-4a58-9ecf-2c9c27b32d62 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:32:45.382041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0f4a3001-0d44-4a58-9ecf-2c9c27b32d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:45.581611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 654/1308] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:32:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:32:45.901704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1ff122-65c1-4afb-9e6b-39976a8f1e9e None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:45.901986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1ff122-65c1-4afb-9e6b-39976a8f1e9e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:45.902516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1ff122-65c1-4afb-9e6b-39976a8f1e9e None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:45.904225 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 655/1309] 10.222.0.77 () {54 vars in 965 bytes} [Mon Aug 29 22:32:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:32:45.911992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cb132c-5be7-4470-acb7-27f6b1730ac2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:45.912453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cb132c-5be7-4470-acb7-27f6b1730ac2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:45.912743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cb132c-5be7-4470-acb7-27f6b1730ac2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:46.015144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f8cb132c-5be7-4470-acb7-27f6b1730ac2 None None] MFA Rules not processed for user `dea13198ef8e45de85b92e79e3ee71b0`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:32:46.017281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f8cb132c-5be7-4470-acb7-27f6b1730ac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:46.050101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:46.146126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 655/1310] 10.222.0.77 () {58 vars in 1062 bytes} [Mon Aug 29 22:32:45 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:32:46.156364 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1ff122-65c1-4afb-9e6b-39976a8f1e9e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:32:46.159639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0d1ff122-65c1-4afb-9e6b-39976a8f1e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:46.336620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1ff122-65c1-4afb-9e6b-39976a8f1e9e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:32:46.459557 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c605732-9d49-4e4a-aa39-dd64c63c5764 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:32:46.460513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c605732-9d49-4e4a-aa39-dd64c63c5764 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:46.460830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c605732-9d49-4e4a-aa39-dd64c63c5764 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:46.461064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c605732-9d49-4e4a-aa39-dd64c63c5764 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:46.462725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2c605732-9d49-4e4a-aa39-dd64c63c5764 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:46.565500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:46.651898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c605732-9d49-4e4a-aa39-dd64c63c5764 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:32:46.653395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c605732-9d49-4e4a-aa39-dd64c63c5764 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:32:46.676062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 656/1311] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:32:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:32:49.041074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07270b18-5823-42c4-9951-cf257a2eba23 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:49.041650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07270b18-5823-42c4-9951-cf257a2eba23 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:49.041969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07270b18-5823-42c4-9951-cf257a2eba23 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:49.044398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 656/1312] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:32:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:32:49.053048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9b27f-f0aa-4fd4-8d93-4836fb1f064d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:49.053578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9b27f-f0aa-4fd4-8d93-4836fb1f064d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:49.053684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9b27f-f0aa-4fd4-8d93-4836fb1f064d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:49.167878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-93d9b27f-f0aa-4fd4-8d93-4836fb1f064d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:32:49.168970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-93d9b27f-f0aa-4fd4-8d93-4836fb1f064d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:49.274390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 657/1313] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:32:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:32:49.560055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07270b18-5823-42c4-9951-cf257a2eba23 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:32:49.561715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-07270b18-5823-42c4-9951-cf257a2eba23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:49.727860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07270b18-5823-42c4-9951-cf257a2eba23 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:32:49.828651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:49.833547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51309c92-5a70-44b8-9717-81abc5e5937c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:32:49.834567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51309c92-5a70-44b8-9717-81abc5e5937c service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:49.835331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51309c92-5a70-44b8-9717-81abc5e5937c service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:49.836015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51309c92-5a70-44b8-9717-81abc5e5937c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:49.838966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-51309c92-5a70-44b8-9717-81abc5e5937c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:49.998230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51309c92-5a70-44b8-9717-81abc5e5937c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:32:50.000078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51309c92-5a70-44b8-9717-81abc5e5937c service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:32:50.021450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 657/1314] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:32:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:32:52.482957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edac62c-9bbe-449a-a748-252816b5db5c None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:52.484256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edac62c-9bbe-449a-a748-252816b5db5c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:52.485061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edac62c-9bbe-449a-a748-252816b5db5c None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:52.487034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 658/1315] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:32:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:32:52.493599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a36433-1b7c-4b63-838b-93e96cd1e0b7 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:52.494146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a36433-1b7c-4b63-838b-93e96cd1e0b7 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:52.494241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a36433-1b7c-4b63-838b-93e96cd1e0b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:52.569124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-73a36433-1b7c-4b63-838b-93e96cd1e0b7 None None] MFA Rules not processed for user `e74c08a75a3a4d6fa1d1523a20e5fbb1`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:32:52.570365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-73a36433-1b7c-4b63-838b-93e96cd1e0b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:52.628340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:52.680614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 658/1316] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:32:52 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:32:52.982015 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556d80c9-6a32-4fa7-88c9-d2844bdd0809 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:52.982301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556d80c9-6a32-4fa7-88c9-d2844bdd0809 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:52.982580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556d80c9-6a32-4fa7-88c9-d2844bdd0809 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:52.984558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 659/1317] 10.222.0.77 () {54 vars in 965 bytes} [Mon Aug 29 22:32:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:32:52.991442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89988b10-07ed-4e10-a4aa-0af4dc3d9c82 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:52.991724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89988b10-07ed-4e10-a4aa-0af4dc3d9c82 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:52.992032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89988b10-07ed-4e10-a4aa-0af4dc3d9c82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:53.077894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-89988b10-07ed-4e10-a4aa-0af4dc3d9c82 None None] MFA Rules not processed for user `dea13198ef8e45de85b92e79e3ee71b0`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:32:53.079048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-89988b10-07ed-4e10-a4aa-0af4dc3d9c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:53.162128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 659/1318] 10.222.0.77 () {58 vars in 1062 bytes} [Mon Aug 29 22:32:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:32:53.172108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556d80c9-6a32-4fa7-88c9-d2844bdd0809 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:32:53.173832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-556d80c9-6a32-4fa7-88c9-d2844bdd0809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:53.184300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:53.316512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556d80c9-6a32-4fa7-88c9-d2844bdd0809 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:32:53.409579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0838526d-23b0-4327-ba3e-32c42c7f241c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:32:53.410163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0838526d-23b0-4327-ba3e-32c42c7f241c service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:53.410490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0838526d-23b0-4327-ba3e-32c42c7f241c service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:53.410840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0838526d-23b0-4327-ba3e-32c42c7f241c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:53.412584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0838526d-23b0-4327-ba3e-32c42c7f241c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:53.485300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:53.549476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0838526d-23b0-4327-ba3e-32c42c7f241c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:32:53.550984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0838526d-23b0-4327-ba3e-32c42c7f241c service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:32:53.572501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 660/1319] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:32:53 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:32:56.540993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015a0d3-0cde-4b80-b287-3e0ad3294f48 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:56.541555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015a0d3-0cde-4b80-b287-3e0ad3294f48 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:56.541555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015a0d3-0cde-4b80-b287-3e0ad3294f48 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:56.542923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 660/1320] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:32:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:32:56.548428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b08c3-f5d0-43a0-beeb-49eb8a3038cb None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:56.549278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b08c3-f5d0-43a0-beeb-49eb8a3038cb None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:56.549278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b08c3-f5d0-43a0-beeb-49eb8a3038cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:56.629495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-b32b08c3-f5d0-43a0-beeb-49eb8a3038cb None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:32:56.630761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b32b08c3-f5d0-43a0-beeb-49eb8a3038cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:56.749132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 661/1321] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:32:56 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:32:57.033691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c015a0d3-0cde-4b80-b287-3e0ad3294f48 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:32:57.035278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c015a0d3-0cde-4b80-b287-3e0ad3294f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:57.090389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:57.152843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c015a0d3-0cde-4b80-b287-3e0ad3294f48 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:32:57.243890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c7c58b-c3a3-4e77-91b2-b2826872e670 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:32:57.244501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7c58b-c3a3-4e77-91b2-b2826872e670 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:32:57.244780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7c58b-c3a3-4e77-91b2-b2826872e670 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:32:57.245056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7c58b-c3a3-4e77-91b2-b2826872e670 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:32:57.246680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-25c7c58b-c3a3-4e77-91b2-b2826872e670 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:32:57.354891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:32:57.370028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c7c58b-c3a3-4e77-91b2-b2826872e670 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:32:57.371562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c7c58b-c3a3-4e77-91b2-b2826872e670 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:32:57.391492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 661/1322] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:32:57 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:00.073496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c933b-78b2-4fe9-a51b-8a4fe4a5b58a None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:00.074088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c933b-78b2-4fe9-a51b-8a4fe4a5b58a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:00.074088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c933b-78b2-4fe9-a51b-8a4fe4a5b58a None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:00.075401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 662/1323] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:00.081498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba959bf-22d9-434f-88f6-aa1be548b195 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:00.082062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba959bf-22d9-434f-88f6-aa1be548b195 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:00.082221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba959bf-22d9-434f-88f6-aa1be548b195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:00.158661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-4ba959bf-22d9-434f-88f6-aa1be548b195 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:00.160324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4ba959bf-22d9-434f-88f6-aa1be548b195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:00.268407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 662/1324] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:33:00.563379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868c933b-78b2-4fe9-a51b-8a4fe4a5b58a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:00.564980 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-868c933b-78b2-4fe9-a51b-8a4fe4a5b58a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:00.603975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:00.718290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868c933b-78b2-4fe9-a51b-8a4fe4a5b58a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:00.804083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51a7d1d-0b6e-4ad3-a851-2a82bbd10429 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:00.804907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a7d1d-0b6e-4ad3-a851-2a82bbd10429 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:00.804907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a7d1d-0b6e-4ad3-a851-2a82bbd10429 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:00.805464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a7d1d-0b6e-4ad3-a851-2a82bbd10429 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:00.806891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a51a7d1d-0b6e-4ad3-a851-2a82bbd10429 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:00.899380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:00.944584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51a7d1d-0b6e-4ad3-a851-2a82bbd10429 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:00.946030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51a7d1d-0b6e-4ad3-a851-2a82bbd10429 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:00.966504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 663/1325] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:02.972519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a680f239-4913-46e2-84d0-7a8f8a8bb055 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:02.973195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a680f239-4913-46e2-84d0-7a8f8a8bb055 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:02.973195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a680f239-4913-46e2-84d0-7a8f8a8bb055 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:02.974425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 663/1326] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:02.979714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce15423-33ae-4c69-b4cc-e4fc2645d6b3 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:02.980315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce15423-33ae-4c69-b4cc-e4fc2645d6b3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:02.980498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce15423-33ae-4c69-b4cc-e4fc2645d6b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:03.064063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3ce15423-33ae-4c69-b4cc-e4fc2645d6b3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:03.065153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3ce15423-33ae-4c69-b4cc-e4fc2645d6b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:03.176891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 664/1327] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:03.490321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a680f239-4913-46e2-84d0-7a8f8a8bb055 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:03.491945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a680f239-4913-46e2-84d0-7a8f8a8bb055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:03.569711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:03.631579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a680f239-4913-46e2-84d0-7a8f8a8bb055 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:03.748109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1e605b-2404-4017-8c1a-e7ec747a144e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:03.749002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e605b-2404-4017-8c1a-e7ec747a144e service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:03.749496 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e605b-2404-4017-8c1a-e7ec747a144e service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:03.749903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e605b-2404-4017-8c1a-e7ec747a144e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:03.751579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-bc1e605b-2404-4017-8c1a-e7ec747a144e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:03.893425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:03.896673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1e605b-2404-4017-8c1a-e7ec747a144e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:03.899781 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1e605b-2404-4017-8c1a-e7ec747a144e service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:03.924402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 664/1328] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:06.081757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036b4cd-bc7a-408c-ab20-af733f6eaf13 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:06.082296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036b4cd-bc7a-408c-ab20-af733f6eaf13 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:06.082296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036b4cd-bc7a-408c-ab20-af733f6eaf13 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:06.083780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 665/1329] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:06.090592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380b021-3405-4f0b-ae52-be2f53fdb99c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:06.091250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380b021-3405-4f0b-ae52-be2f53fdb99c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:06.091490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380b021-3405-4f0b-ae52-be2f53fdb99c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:06.165484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-0380b021-3405-4f0b-ae52-be2f53fdb99c None None] MFA Rules not processed for user `e74c08a75a3a4d6fa1d1523a20e5fbb1`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:06.166718 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0380b021-3405-4f0b-ae52-be2f53fdb99c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:06.303028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 665/1330] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:06 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:06.611170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2036b4cd-bc7a-408c-ab20-af733f6eaf13 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:06.614642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2036b4cd-bc7a-408c-ab20-af733f6eaf13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:06.648896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:06.732007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2036b4cd-bc7a-408c-ab20-af733f6eaf13 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:06.824391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da233cb-752f-4e92-9927-468f0853d661 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:06.825214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da233cb-752f-4e92-9927-468f0853d661 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:06.825618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da233cb-752f-4e92-9927-468f0853d661 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:06.825618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da233cb-752f-4e92-9927-468f0853d661 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:06.827289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5da233cb-752f-4e92-9927-468f0853d661 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:06.930072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:06.974925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da233cb-752f-4e92-9927-468f0853d661 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:06.977162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da233cb-752f-4e92-9927-468f0853d661 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:07.004941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 666/1331] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:06 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:09.321461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a6e35-e56e-4e97-a49a-e5f86f9c851f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:09.322032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a6e35-e56e-4e97-a49a-e5f86f9c851f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:09.322032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a6e35-e56e-4e97-a49a-e5f86f9c851f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:09.323924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 666/1332] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:09.330256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1bd3fb-0b37-4e10-bf6a-a25a28db7c77 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:09.330929 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1bd3fb-0b37-4e10-bf6a-a25a28db7c77 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:09.330929 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1bd3fb-0b37-4e10-bf6a-a25a28db7c77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:09.409734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8e1bd3fb-0b37-4e10-bf6a-a25a28db7c77 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:09.411139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8e1bd3fb-0b37-4e10-bf6a-a25a28db7c77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:09.512359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 667/1333] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:09.799007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5a6e35-e56e-4e97-a49a-e5f86f9c851f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:09.800763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7e5a6e35-e56e-4e97-a49a-e5f86f9c851f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:09.886187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:09.927334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5a6e35-e56e-4e97-a49a-e5f86f9c851f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:10.015843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d27985f-77cc-4384-8237-bf1ec337a691 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:10.016812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d27985f-77cc-4384-8237-bf1ec337a691 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:10.017164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d27985f-77cc-4384-8237-bf1ec337a691 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:10.017464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d27985f-77cc-4384-8237-bf1ec337a691 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:10.019726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4d27985f-77cc-4384-8237-bf1ec337a691 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:10.146330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d27985f-77cc-4384-8237-bf1ec337a691 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:10.147717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d27985f-77cc-4384-8237-bf1ec337a691 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:10.158933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:10.174852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 667/1334] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:12.257374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760c1ce9-88f9-49fe-8d6c-413c29197519 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:12.258073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760c1ce9-88f9-49fe-8d6c-413c29197519 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:12.258073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760c1ce9-88f9-49fe-8d6c-413c29197519 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:12.259299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 668/1335] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:12.265084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7abe4e-00a6-4b72-b4b7-240b7e6cc4e0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:12.265588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7abe4e-00a6-4b72-b4b7-240b7e6cc4e0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:12.265699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7abe4e-00a6-4b72-b4b7-240b7e6cc4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:12.336573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1e7abe4e-00a6-4b72-b4b7-240b7e6cc4e0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:12.337806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1e7abe4e-00a6-4b72-b4b7-240b7e6cc4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:12.443456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 668/1336] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:12.727791 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760c1ce9-88f9-49fe-8d6c-413c29197519 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:12.729349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-760c1ce9-88f9-49fe-8d6c-413c29197519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:12.776205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:12.859544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760c1ce9-88f9-49fe-8d6c-413c29197519 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:12.940815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5860fea4-f97a-412c-92bc-1a021b456cea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:12.941425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860fea4-f97a-412c-92bc-1a021b456cea service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:12.941708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860fea4-f97a-412c-92bc-1a021b456cea service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:12.941983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860fea4-f97a-412c-92bc-1a021b456cea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:12.943676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5860fea4-f97a-412c-92bc-1a021b456cea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:13.037210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:13.070385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5860fea4-f97a-412c-92bc-1a021b456cea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:13.071888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5860fea4-f97a-412c-92bc-1a021b456cea service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:13.090438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 669/1337] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:15.498387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd8203-8b74-4cea-8fa9-3d4e9f8455c5 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:15.498990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd8203-8b74-4cea-8fa9-3d4e9f8455c5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:15.498990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd8203-8b74-4cea-8fa9-3d4e9f8455c5 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:15.500343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 669/1338] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:15.506021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958fe02-0b02-424e-bc3b-520b5985d41a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:15.506587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958fe02-0b02-424e-bc3b-520b5985d41a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:15.506587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958fe02-0b02-424e-bc3b-520b5985d41a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:15.582191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8958fe02-0b02-424e-bc3b-520b5985d41a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:15.583456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8958fe02-0b02-424e-bc3b-520b5985d41a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:15.707557 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 670/1339] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:15 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:15.997718 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edd8203-8b74-4cea-8fa9-3d4e9f8455c5 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:15.999334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2edd8203-8b74-4cea-8fa9-3d4e9f8455c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:16.089876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:16.120615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edd8203-8b74-4cea-8fa9-3d4e9f8455c5 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:16.197686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d146dd-3265-4208-a8a7-ae385539c631 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:16.198288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d146dd-3265-4208-a8a7-ae385539c631 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:16.198566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d146dd-3265-4208-a8a7-ae385539c631 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:16.198866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d146dd-3265-4208-a8a7-ae385539c631 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:16.200491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-16d146dd-3265-4208-a8a7-ae385539c631 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:16.325782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d146dd-3265-4208-a8a7-ae385539c631 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:16.327269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d146dd-3265-4208-a8a7-ae385539c631 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:16.345570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 670/1340] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:15 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:33:20.390231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431847a1-051e-4743-b42e-83e7fcf34f0b None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:20.390797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431847a1-051e-4743-b42e-83e7fcf34f0b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:20.390896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431847a1-051e-4743-b42e-83e7fcf34f0b None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:20.392193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 671/1341] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:33:20.398010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc61018-c5b5-4524-b822-334a89988250 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:20.398481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc61018-c5b5-4524-b822-334a89988250 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:20.398692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc61018-c5b5-4524-b822-334a89988250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:20.404498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:20.471906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-adc61018-c5b5-4524-b822-334a89988250 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:20.473144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-adc61018-c5b5-4524-b822-334a89988250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:20.582435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 671/1342] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:33:20.871986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431847a1-051e-4743-b42e-83e7fcf34f0b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:20.873539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-431847a1-051e-4743-b42e-83e7fcf34f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:20.945718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:21.014037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431847a1-051e-4743-b42e-83e7fcf34f0b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:21.105511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2756d5-beb2-47d7-8e04-e205caf7f71d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:21.106125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2756d5-beb2-47d7-8e04-e205caf7f71d service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:21.106554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2756d5-beb2-47d7-8e04-e205caf7f71d service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:21.106861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2756d5-beb2-47d7-8e04-e205caf7f71d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:21.108638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9d2756d5-beb2-47d7-8e04-e205caf7f71d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:21.223258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:21.254700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2756d5-beb2-47d7-8e04-e205caf7f71d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:21.256208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2756d5-beb2-47d7-8e04-e205caf7f71d service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:21.277586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 672/1343] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:24.179504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d4752-42bf-4518-863a-473c15b572c5 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:24.180069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d4752-42bf-4518-863a-473c15b572c5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:24.180183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d4752-42bf-4518-863a-473c15b572c5 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:24.181523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 672/1344] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:24.186941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2bdec-3045-4ff9-abbb-007c7340b5fb None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:24.187396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2bdec-3045-4ff9-abbb-007c7340b5fb None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:24.187630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2bdec-3045-4ff9-abbb-007c7340b5fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:24.267777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-daa2bdec-3045-4ff9-abbb-007c7340b5fb None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:24.269074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-daa2bdec-3045-4ff9-abbb-007c7340b5fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:24.375127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 673/1345] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:24 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:33:24.661879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1d4752-42bf-4518-863a-473c15b572c5 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:24.663472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0a1d4752-42bf-4518-863a-473c15b572c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:24.766788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:24.791319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1d4752-42bf-4518-863a-473c15b572c5 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:24.876333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b98172-559d-4688-8d5a-d2acb86f72d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:24.876940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b98172-559d-4688-8d5a-d2acb86f72d2 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:24.877215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b98172-559d-4688-8d5a-d2acb86f72d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:24.877486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b98172-559d-4688-8d5a-d2acb86f72d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:24.879142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-47b98172-559d-4688-8d5a-d2acb86f72d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:25.010689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b98172-559d-4688-8d5a-d2acb86f72d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:25.012279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b98172-559d-4688-8d5a-d2acb86f72d2 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:25.034137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 673/1346] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:24 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:33:29.309833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9781e169-66cc-4e08-a673-8838154edd59 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:29.310381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9781e169-66cc-4e08-a673-8838154edd59 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:29.310509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9781e169-66cc-4e08-a673-8838154edd59 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:29.312003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 674/1347] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:29.317632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e933e6cf-6eb8-4199-a60e-f99b89c2d61e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:29.318624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e933e6cf-6eb8-4199-a60e-f99b89c2d61e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:29.318624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e933e6cf-6eb8-4199-a60e-f99b89c2d61e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:29.335523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:29.400240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e933e6cf-6eb8-4199-a60e-f99b89c2d61e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:29.401463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e933e6cf-6eb8-4199-a60e-f99b89c2d61e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:29.521361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 674/1348] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:29.818199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9781e169-66cc-4e08-a673-8838154edd59 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:29.819886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9781e169-66cc-4e08-a673-8838154edd59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:29.880739 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:29.927862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9781e169-66cc-4e08-a673-8838154edd59 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:30.009202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5067726c-97bd-487c-8de7-01c731c2c34c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:30.009814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5067726c-97bd-487c-8de7-01c731c2c34c service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:30.010092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5067726c-97bd-487c-8de7-01c731c2c34c service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:30.010594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5067726c-97bd-487c-8de7-01c731c2c34c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:30.012079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5067726c-97bd-487c-8de7-01c731c2c34c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:30.148874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:30.153875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5067726c-97bd-487c-8de7-01c731c2c34c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:30.155379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5067726c-97bd-487c-8de7-01c731c2c34c service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:30.179792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 675/1349] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:35.745891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b786b88-054b-4f0b-85cd-b29f5621c2e6 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:35.746464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b786b88-054b-4f0b-85cd-b29f5621c2e6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:35.746464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b786b88-054b-4f0b-85cd-b29f5621c2e6 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:35.751785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 675/1350] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:33:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:35.758565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b438c-1752-4fdd-98fa-a3a301447c83 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:35.763159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b438c-1752-4fdd-98fa-a3a301447c83 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:35.763524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b438c-1752-4fdd-98fa-a3a301447c83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:35.884669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-543b438c-1752-4fdd-98fa-a3a301447c83 None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:35.886235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-543b438c-1752-4fdd-98fa-a3a301447c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:36.132403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 676/1351] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:33:35 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 374 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:33:39.519457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=59 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:41.060766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9302aa18-3575-49d0-8639-753bcf05a531 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:41.061412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9302aa18-3575-49d0-8639-753bcf05a531 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:41.061412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9302aa18-3575-49d0-8639-753bcf05a531 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:41.062845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 676/1352] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:33:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:33:41.069800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8bbac2-4356-40cd-8bac-3dfc7ca5d66a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:41.070325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8bbac2-4356-40cd-8bac-3dfc7ca5d66a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:41.070577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8bbac2-4356-40cd-8bac-3dfc7ca5d66a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:41.138985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:41.184252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-4c8bbac2-4356-40cd-8bac-3dfc7ca5d66a None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:41.186026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4c8bbac2-4356-40cd-8bac-3dfc7ca5d66a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:41.370637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 677/1353] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:33:41 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:43.060490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad7523-346d-4ec9-a839-9e2075518d34 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:43.061233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad7523-346d-4ec9-a839-9e2075518d34 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:43.061233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad7523-346d-4ec9-a839-9e2075518d34 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:43.062587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 677/1354] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:33:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:43.067926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce276652-cb45-4368-a107-4cb84aa8a9a7 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:43.068411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce276652-cb45-4368-a107-4cb84aa8a9a7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:43.068538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce276652-cb45-4368-a107-4cb84aa8a9a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:43.150249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-ce276652-cb45-4368-a107-4cb84aa8a9a7 None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:43.151411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ce276652-cb45-4368-a107-4cb84aa8a9a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:43.205350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:43.285373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 678/1355] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:33:43 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:33:46.574814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4868cb18-b969-464a-834d-ca1a808d6213 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:46.575402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4868cb18-b969-464a-834d-ca1a808d6213 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:46.575402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4868cb18-b969-464a-834d-ca1a808d6213 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:46.576937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 678/1356] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:33:46.582479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca77fbdd-d4bb-4ccf-85a2-a18ad8d80398 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:46.582962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca77fbdd-d4bb-4ccf-85a2-a18ad8d80398 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:46.583161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca77fbdd-d4bb-4ccf-85a2-a18ad8d80398 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:46.665046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-ca77fbdd-d4bb-4ccf-85a2-a18ad8d80398 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:46.666782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ca77fbdd-d4bb-4ccf-85a2-a18ad8d80398 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:46.799192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 679/1357] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:47.119586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c034a7-e2af-4248-9957-31a9121fae02 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:47.120327 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c034a7-e2af-4248-9957-31a9121fae02 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:47.120819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c034a7-e2af-4248-9957-31a9121fae02 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:47.122457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 679/1358] 10.222.0.77 () {54 vars in 962 bytes} [Mon Aug 29 22:33:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:47.135408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3312764d-e0c4-40fb-846b-eb87d0049464 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:47.135778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3312764d-e0c4-40fb-846b-eb87d0049464 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:47.136204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3312764d-e0c4-40fb-846b-eb87d0049464 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:47.154968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:47.233620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3312764d-e0c4-40fb-846b-eb87d0049464 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:47.234948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3312764d-e0c4-40fb-846b-eb87d0049464 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:47.371173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 680/1359] 10.222.0.77 () {58 vars in 1059 bytes} [Mon Aug 29 22:33:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:47.380941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c034a7-e2af-4248-9957-31a9121fae02 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:47.382649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-16c034a7-e2af-4248-9957-31a9121fae02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:47.561460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c034a7-e2af-4248-9957-31a9121fae02 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:47.656398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:47.712610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fc9a58-9300-40a7-802f-29d215644e7e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:47.713402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fc9a58-9300-40a7-802f-29d215644e7e service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:47.713787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fc9a58-9300-40a7-802f-29d215644e7e service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:47.714166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fc9a58-9300-40a7-802f-29d215644e7e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:47.716022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a5fc9a58-9300-40a7-802f-29d215644e7e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:47.826698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fc9a58-9300-40a7-802f-29d215644e7e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:47.828798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fc9a58-9300-40a7-802f-29d215644e7e service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:47.853652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 680/1360] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:50.127800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5f628a-e3d4-4008-9184-2775537f18d8 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:50.128348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5f628a-e3d4-4008-9184-2775537f18d8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:50.128639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5f628a-e3d4-4008-9184-2775537f18d8 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:50.130601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 681/1361] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:50.139195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c1cbc5-6ae9-400f-b50a-e2a41dbc621b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:50.140026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c1cbc5-6ae9-400f-b50a-e2a41dbc621b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:50.140026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c1cbc5-6ae9-400f-b50a-e2a41dbc621b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:50.253867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e9c1cbc5-6ae9-400f-b50a-e2a41dbc621b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:50.255920 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e9c1cbc5-6ae9-400f-b50a-e2a41dbc621b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:50.308762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:50.396201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 681/1362] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:50.696549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5f628a-e3d4-4008-9184-2775537f18d8 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:50.699259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0c5f628a-e3d4-4008-9184-2775537f18d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:50.818124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:50.923552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5f628a-e3d4-4008-9184-2775537f18d8 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:51.078968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a3f329-c2a9-4c20-89bd-336164549701 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:51.079605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3f329-c2a9-4c20-89bd-336164549701 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:51.079892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3f329-c2a9-4c20-89bd-336164549701 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:51.080167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3f329-c2a9-4c20-89bd-336164549701 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:51.081979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d7a3f329-c2a9-4c20-89bd-336164549701 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:51.118886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:51.209103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a3f329-c2a9-4c20-89bd-336164549701 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:51.211223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a3f329-c2a9-4c20-89bd-336164549701 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:51.235191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 682/1363] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 539 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:53.604563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317d58fa-479a-4038-9e5c-e4bfed6959ac None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:53.605255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317d58fa-479a-4038-9e5c-e4bfed6959ac None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:53.605255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317d58fa-479a-4038-9e5c-e4bfed6959ac None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:53.606932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 682/1364] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:53.613492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645bb888-5029-4f64-a4ba-66ccf4824fef None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:53.614028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645bb888-5029-4f64-a4ba-66ccf4824fef None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:53.614028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645bb888-5029-4f64-a4ba-66ccf4824fef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:53.700030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-645bb888-5029-4f64-a4ba-66ccf4824fef None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:53.701792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-645bb888-5029-4f64-a4ba-66ccf4824fef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:53.811413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:53.822491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 683/1365] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:54.115903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d64d7-5684-4f1b-aea2-b955f03fc508 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:54.116282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d64d7-5684-4f1b-aea2-b955f03fc508 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:54.116480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d64d7-5684-4f1b-aea2-b955f03fc508 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:54.118043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 683/1366] 10.222.0.77 () {54 vars in 962 bytes} [Mon Aug 29 22:33:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:54.125989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ecf24b-626c-4eb9-bd24-0930a3443d87 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:54.126284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ecf24b-626c-4eb9-bd24-0930a3443d87 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:54.126904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ecf24b-626c-4eb9-bd24-0930a3443d87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:54.240757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e5ecf24b-626c-4eb9-bd24-0930a3443d87 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:54.242114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e5ecf24b-626c-4eb9-bd24-0930a3443d87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:54.417823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 684/1367] 10.222.0.77 () {58 vars in 1059 bytes} [Mon Aug 29 22:33:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:54.427382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59d64d7-5684-4f1b-aea2-b955f03fc508 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:54.429392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e59d64d7-5684-4f1b-aea2-b955f03fc508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:54.640521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59d64d7-5684-4f1b-aea2-b955f03fc508 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:54.668601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:54.782157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aec2634-066e-46e3-8959-6f9e3ca93507 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:54.782814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aec2634-066e-46e3-8959-6f9e3ca93507 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:54.783110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aec2634-066e-46e3-8959-6f9e3ca93507 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:54.786881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aec2634-066e-46e3-8959-6f9e3ca93507 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:54.786881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8aec2634-066e-46e3-8959-6f9e3ca93507 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:54.892157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aec2634-066e-46e3-8959-6f9e3ca93507 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:54.894777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aec2634-066e-46e3-8959-6f9e3ca93507 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:54.916025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 684/1368] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:55.414642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc97d486-150a-4e77-9f25-bfd9e0d95de3 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:55.415245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc97d486-150a-4e77-9f25-bfd9e0d95de3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:55.415245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc97d486-150a-4e77-9f25-bfd9e0d95de3 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:55.416849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 685/1369] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:55.422781 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaf18bc-d426-40dc-9d50-71e67e5b62c6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:55.423171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaf18bc-d426-40dc-9d50-71e67e5b62c6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:55.423588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaf18bc-d426-40dc-9d50-71e67e5b62c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:55.460465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:55.508856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-edaf18bc-d426-40dc-9d50-71e67e5b62c6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:55.510291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-edaf18bc-d426-40dc-9d50-71e67e5b62c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:55.629880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 685/1370] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:56.260227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a52697-2c23-47e5-8e3b-e6b4dfeaac56 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:56.260227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a52697-2c23-47e5-8e3b-e6b4dfeaac56 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:56.260227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a52697-2c23-47e5-8e3b-e6b4dfeaac56 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:56.270976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 686/1371] 10.222.0.77 () {54 vars in 964 bytes} [Mon Aug 29 22:33:56 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:56.278782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce249e5-28cd-4e9b-bfc2-8ad42c4fcc83 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:56.279489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce249e5-28cd-4e9b-bfc2-8ad42c4fcc83 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:56.279489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce249e5-28cd-4e9b-bfc2-8ad42c4fcc83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:56.390909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-4ce249e5-28cd-4e9b-bfc2-8ad42c4fcc83 None None] MFA Rules not processed for user `7964b4109f9a4ac6a68d90366489557c`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:56.392922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4ce249e5-28cd-4e9b-bfc2-8ad42c4fcc83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:56.444036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:56.481129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 686/1372] 10.222.0.77 () {58 vars in 1061 bytes} [Mon Aug 29 22:33:56 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:56.489414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a52697-2c23-47e5-8e3b-e6b4dfeaac56 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:56.491065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-58a52697-2c23-47e5-8e3b-e6b4dfeaac56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:56.553076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:56.616381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a52697-2c23-47e5-8e3b-e6b4dfeaac56 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:56.727358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be270c4-ad0b-4c60-a80c-31d836b4d22c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:56.729149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be270c4-ad0b-4c60-a80c-31d836b4d22c service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:56.729756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be270c4-ad0b-4c60-a80c-31d836b4d22c service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:56.729924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be270c4-ad0b-4c60-a80c-31d836b4d22c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:56.731643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8be270c4-ad0b-4c60-a80c-31d836b4d22c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:56.897427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:56.912823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be270c4-ad0b-4c60-a80c-31d836b4d22c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:56.916657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be270c4-ad0b-4c60-a80c-31d836b4d22c service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:56.941103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 687/1373] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:33:56.991066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39bcd7c-bb40-4246-8856-112a4db15721 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:56.991367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39bcd7c-bb40-4246-8856-112a4db15721 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:56.991659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39bcd7c-bb40-4246-8856-112a4db15721 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:56.992954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 687/1374] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:56.998636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43602078-f10e-401e-9200-ca22f2ead3a5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:56.998967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43602078-f10e-401e-9200-ca22f2ead3a5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:56.999243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43602078-f10e-401e-9200-ca22f2ead3a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:57.093023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-43602078-f10e-401e-9200-ca22f2ead3a5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:57.094388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-43602078-f10e-401e-9200-ca22f2ead3a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:57.211121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 688/1375] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:33:57.512242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39bcd7c-bb40-4246-8856-112a4db15721 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:33:57.514603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b39bcd7c-bb40-4246-8856-112a4db15721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:57.687192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39bcd7c-bb40-4246-8856-112a4db15721 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:33:57.742314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:33:57.807989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0ca54-79e4-47af-9aa6-9601facecddc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:33:57.808511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0ca54-79e4-47af-9aa6-9601facecddc service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:57.808864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0ca54-79e4-47af-9aa6-9601facecddc service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:57.809146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0ca54-79e4-47af-9aa6-9601facecddc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:57.810887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-55b0ca54-79e4-47af-9aa6-9601facecddc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:33:57.905488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b0ca54-79e4-47af-9aa6-9601facecddc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:33:57.907179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b0ca54-79e4-47af-9aa6-9601facecddc service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:33:57.927339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 688/1376] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:33:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:33:59.914266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8853c5-979b-4c93-a792-8c471c077018 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:59.914266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8853c5-979b-4c93-a792-8c471c077018 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:59.914266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8853c5-979b-4c93-a792-8c471c077018 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:59.915066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 689/1377] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:33:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:33:59.920348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd657ab-eb9f-4ae1-b533-7dcdb98d3914 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:33:59.920806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd657ab-eb9f-4ae1-b533-7dcdb98d3914 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:33:59.920910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd657ab-eb9f-4ae1-b533-7dcdb98d3914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:33:59.993583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7fd657ab-eb9f-4ae1-b533-7dcdb98d3914 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:33:59.994846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7fd657ab-eb9f-4ae1-b533-7dcdb98d3914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:00.008695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:00.147371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 689/1378] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:33:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:00.439955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8853c5-979b-4c93-a792-8c471c077018 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:00.441569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0d8853c5-979b-4c93-a792-8c471c077018 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:00.542573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:00.668928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8853c5-979b-4c93-a792-8c471c077018 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:00.835453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15f93f2-242b-4aca-bfdc-f82ccb9e44c5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:00.836223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f93f2-242b-4aca-bfdc-f82ccb9e44c5 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:00.836699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f93f2-242b-4aca-bfdc-f82ccb9e44c5 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:00.837169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f93f2-242b-4aca-bfdc-f82ccb9e44c5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:00.839642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c15f93f2-242b-4aca-bfdc-f82ccb9e44c5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:00.863540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:00.946342 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15f93f2-242b-4aca-bfdc-f82ccb9e44c5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:00.948023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15f93f2-242b-4aca-bfdc-f82ccb9e44c5 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:00.969024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 690/1379] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 529 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:03.046947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78d93b-5eb7-4f4e-94d3-569055946a36 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:03.048090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78d93b-5eb7-4f4e-94d3-569055946a36 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:03.048182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78d93b-5eb7-4f4e-94d3-569055946a36 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:03.049680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 690/1380] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:34:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:03.054613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b912c-be2a-4bd0-a775-df287077e26d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:03.055279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b912c-be2a-4bd0-a775-df287077e26d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:03.055279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b912c-be2a-4bd0-a775-df287077e26d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:03.137438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-ef4b912c-be2a-4bd0-a775-df287077e26d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:03.138659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ef4b912c-be2a-4bd0-a775-df287077e26d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:03.243002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:03.250965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 691/1381] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:34:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:03.546522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e78d93b-5eb7-4f4e-94d3-569055946a36 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:03.549156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8e78d93b-5eb7-4f4e-94d3-569055946a36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:03.734956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:03.748071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e78d93b-5eb7-4f4e-94d3-569055946a36 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:03.908635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05781efe-1aa6-49a7-8832-fd1c31991b90 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:03.909793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05781efe-1aa6-49a7-8832-fd1c31991b90 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:03.910974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05781efe-1aa6-49a7-8832-fd1c31991b90 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:03.911623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05781efe-1aa6-49a7-8832-fd1c31991b90 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:03.914686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-05781efe-1aa6-49a7-8832-fd1c31991b90 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:04.034657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05781efe-1aa6-49a7-8832-fd1c31991b90 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:04.036463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05781efe-1aa6-49a7-8832-fd1c31991b90 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:04.048154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:04.058105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 691/1382] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 512 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:06.164764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da0ede-46b5-40bd-8045-0fa6d9a1fd5f None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:06.165907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da0ede-46b5-40bd-8045-0fa6d9a1fd5f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:06.165907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da0ede-46b5-40bd-8045-0fa6d9a1fd5f None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:06.167673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 692/1383] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:34:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:34:06.175013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf07430-35ab-4a4f-b081-e26bf2cb7d26 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:06.175749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf07430-35ab-4a4f-b081-e26bf2cb7d26 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:06.176205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf07430-35ab-4a4f-b081-e26bf2cb7d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:06.270216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-0cf07430-35ab-4a4f-b081-e26bf2cb7d26 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:06.271547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0cf07430-35ab-4a4f-b081-e26bf2cb7d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:06.386589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 692/1384] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:34:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:06.690576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da0ede-46b5-40bd-8045-0fa6d9a1fd5f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:06.692514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-97da0ede-46b5-40bd-8045-0fa6d9a1fd5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:06.939610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da0ede-46b5-40bd-8045-0fa6d9a1fd5f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:07.042610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:07.096261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e789f3-08d3-4bc9-9948-ff0caab78439 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:07.096837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e789f3-08d3-4bc9-9948-ff0caab78439 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:07.097138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e789f3-08d3-4bc9-9948-ff0caab78439 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:07.097439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e789f3-08d3-4bc9-9948-ff0caab78439 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:07.099149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-46e789f3-08d3-4bc9-9948-ff0caab78439 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:07.209605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e789f3-08d3-4bc9-9948-ff0caab78439 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:07.211485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e789f3-08d3-4bc9-9948-ff0caab78439 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:07.238482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 693/1385] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 549 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:34:09.356987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728824bb-525d-46dc-b209-415fe4ac2abd None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:09.357588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728824bb-525d-46dc-b209-415fe4ac2abd None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:09.357588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728824bb-525d-46dc-b209-415fe4ac2abd None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:09.359537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 693/1386] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:34:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:09.366003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf9990-e692-4def-9b42-e33ac1487a5e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:09.367024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf9990-e692-4def-9b42-e33ac1487a5e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:09.367024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf9990-e692-4def-9b42-e33ac1487a5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:09.449391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-04cf9990-e692-4def-9b42-e33ac1487a5e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:09.450852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-04cf9990-e692-4def-9b42-e33ac1487a5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:09.504262 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:09.603268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 694/1387] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:34:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:09.949252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728824bb-525d-46dc-b209-415fe4ac2abd None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:09.951184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-728824bb-525d-46dc-b209-415fe4ac2abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:10.093933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:10.209023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728824bb-525d-46dc-b209-415fe4ac2abd None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:10.380366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79c5ad5-7f81-4873-9c88-17166dfaf648 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:10.381279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c5ad5-7f81-4873-9c88-17166dfaf648 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:10.381279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c5ad5-7f81-4873-9c88-17166dfaf648 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:10.381491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c5ad5-7f81-4873-9c88-17166dfaf648 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:10.383177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e79c5ad5-7f81-4873-9c88-17166dfaf648 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:10.427174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:10.490364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79c5ad5-7f81-4873-9c88-17166dfaf648 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:10.492058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79c5ad5-7f81-4873-9c88-17166dfaf648 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:10.512940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 694/1388] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 564 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:12.592541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d631bf39-ca1c-4664-b7b8-e239eca39892 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:12.593252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d631bf39-ca1c-4664-b7b8-e239eca39892 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:12.593252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d631bf39-ca1c-4664-b7b8-e239eca39892 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:12.594660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 695/1389] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:34:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:34:12.601159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1179bdda-9d1b-4403-957e-25dcb403d748 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:12.602006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1179bdda-9d1b-4403-957e-25dcb403d748 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:12.602183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1179bdda-9d1b-4403-957e-25dcb403d748 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:12.604433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c6542-9b9f-4950-a60a-edb30a6cab78 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:12.604721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c6542-9b9f-4950-a60a-edb30a6cab78 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:12.605114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c6542-9b9f-4950-a60a-edb30a6cab78 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:12.606465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 696/1390] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:34:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:34:12.613212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c31c6-535e-4d49-8e6f-12f8390eb0be None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:12.613550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c31c6-535e-4d49-8e6f-12f8390eb0be None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:12.613550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c31c6-535e-4d49-8e6f-12f8390eb0be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:12.706244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1179bdda-9d1b-4403-957e-25dcb403d748 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:12.706781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-9f9c31c6-535e-4d49-8e6f-12f8390eb0be None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:12.707957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1179bdda-9d1b-4403-957e-25dcb403d748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:12.708158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9f9c31c6-535e-4d49-8e6f-12f8390eb0be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:12.833237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 697/1391] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:34:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:12.844249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 695/1392] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:34:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:13.137863 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9c31c6-535e-4d49-8e6f-12f8390eb0be None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:13.139177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9f9c31c6-535e-4d49-8e6f-12f8390eb0be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:13.155492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88ad4d-c231-4b9a-b4b9-223d5d671c4a None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:13.155867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:13.155977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88ad4d-c231-4b9a-b4b9-223d5d671c4a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:13.156234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88ad4d-c231-4b9a-b4b9-223d5d671c4a None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:13.157952 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 696/1393] 10.222.0.77 () {54 vars in 967 bytes} [Mon Aug 29 22:34:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:34:13.162974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c7dcfd-ae39-45a1-8d7f-0f65154e8cb9 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:13.163256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c7dcfd-ae39-45a1-8d7f-0f65154e8cb9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:13.163529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c7dcfd-ae39-45a1-8d7f-0f65154e8cb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:13.179830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:13.243825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-50c7dcfd-ae39-45a1-8d7f-0f65154e8cb9 None None] MFA Rules not processed for user `c1e61f3604fd4a23ae5f1e7e62ae21ae`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:13.245048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-50c7dcfd-ae39-45a1-8d7f-0f65154e8cb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:13.261630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9c31c6-535e-4d49-8e6f-12f8390eb0be None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:13.322200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 697/1394] 10.222.0.77 () {58 vars in 1064 bytes} [Mon Aug 29 22:34:13 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:13.329888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c7dcfd-ae39-45a1-8d7f-0f65154e8cb9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:13.337518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c7dcfd-ae39-45a1-8d7f-0f65154e8cb9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:13.383954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f34db-35dd-4d38-9af0-4373fbcf6456 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:13.384477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f34db-35dd-4d38-9af0-4373fbcf6456 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:13.384756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f34db-35dd-4d38-9af0-4373fbcf6456 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:13.385029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f34db-35dd-4d38-9af0-4373fbcf6456 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:13.386656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-be0f34db-35dd-4d38-9af0-4373fbcf6456 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:13.397292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:13.418218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a12b872-a3c7-4dd5-a27a-949b257fb277 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:13.418867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12b872-a3c7-4dd5-a27a-949b257fb277 service glance] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:13.419144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12b872-a3c7-4dd5-a27a-949b257fb277 service glance] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:13.419410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12b872-a3c7-4dd5-a27a-949b257fb277 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:13.421132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7a12b872-a3c7-4dd5-a27a-949b257fb277 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:13.451030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:13.482217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0f34db-35dd-4d38-9af0-4373fbcf6456 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:13.484042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0f34db-35dd-4d38-9af0-4373fbcf6456 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:13.502533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 698/1395] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:13.537676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a12b872-a3c7-4dd5-a27a-949b257fb277 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:13.539265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a12b872-a3c7-4dd5-a27a-949b257fb277 service glance] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:13.558156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 698/1396] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:34:14.318268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b95c38e-a98d-4eaf-a73b-7761c2f342cb None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:14.319111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b95c38e-a98d-4eaf-a73b-7761c2f342cb None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:14.319305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b95c38e-a98d-4eaf-a73b-7761c2f342cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:14.411033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2b95c38e-a98d-4eaf-a73b-7761c2f342cb None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:14.412873 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2b95c38e-a98d-4eaf-a73b-7761c2f342cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:14.513732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:14.518553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 699/1397] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:34:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:14.569283 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1214e634-88fb-4aba-93f6-4127becd40b6 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:14.569838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1214e634-88fb-4aba-93f6-4127becd40b6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:14.569973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1214e634-88fb-4aba-93f6-4127becd40b6 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:14.571412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 699/1398] 10.222.0.77 () {54 vars in 943 bytes} [Mon Aug 29 22:34:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:14.579732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558fa11-3232-4f48-96d2-d02e9e8400c0 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:14.579993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558fa11-3232-4f48-96d2-d02e9e8400c0 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:14.580275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558fa11-3232-4f48-96d2-d02e9e8400c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:14.694934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7558fa11-3232-4f48-96d2-d02e9e8400c0 None None] MFA Rules not processed for user `03361f4b51d34cf28c4b5451c3b1f881`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:14.696243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7558fa11-3232-4f48-96d2-d02e9e8400c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:14.832162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 700/1399] 10.222.0.77 () {58 vars in 1040 bytes} [Mon Aug 29 22:34:14 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:14.842379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1214e634-88fb-4aba-93f6-4127becd40b6 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:14.844763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1214e634-88fb-4aba-93f6-4127becd40b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:15.004043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:15.006407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1214e634-88fb-4aba-93f6-4127becd40b6 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:15.139923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07143534-0561-4556-830f-0908fdd606f3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:15.140457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07143534-0561-4556-830f-0908fdd606f3 service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:15.140735 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07143534-0561-4556-830f-0908fdd606f3 service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:15.141004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07143534-0561-4556-830f-0908fdd606f3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:15.142835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-07143534-0561-4556-830f-0908fdd606f3 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:15.241010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07143534-0561-4556-830f-0908fdd606f3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:15.242611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07143534-0561-4556-830f-0908fdd606f3 service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:15.252465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 700/1400] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:34:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:34:15.504116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5177bf-d9c7-4a08-a2be-601e1e91daff None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:15.504662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5177bf-d9c7-4a08-a2be-601e1e91daff None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:15.504662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5177bf-d9c7-4a08-a2be-601e1e91daff None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:15.506146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 701/1401] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:34:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:15.511583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc0f213-7cf0-40d8-b047-c521d3c719ec None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:15.511871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc0f213-7cf0-40d8-b047-c521d3c719ec None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:15.512144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc0f213-7cf0-40d8-b047-c521d3c719ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:15.559769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:15.595655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1dc0f213-7cf0-40d8-b047-c521d3c719ec None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:15.597377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1dc0f213-7cf0-40d8-b047-c521d3c719ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:15.735228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 701/1402] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:34:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:16.036119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5177bf-d9c7-4a08-a2be-601e1e91daff None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:16.037686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ea5177bf-d9c7-4a08-a2be-601e1e91daff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:16.118413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:16.264594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5177bf-d9c7-4a08-a2be-601e1e91daff None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:16.471573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8f0840-d4d1-4a95-bc00-13e2f97c6507 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:16.472352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f0840-d4d1-4a95-bc00-13e2f97c6507 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:16.472807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f0840-d4d1-4a95-bc00-13e2f97c6507 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:16.473163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f0840-d4d1-4a95-bc00-13e2f97c6507 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:16.475076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ed8f0840-d4d1-4a95-bc00-13e2f97c6507 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:16.499356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:16.595554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8f0840-d4d1-4a95-bc00-13e2f97c6507 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:16.598610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8f0840-d4d1-4a95-bc00-13e2f97c6507 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:16.636246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 702/1403] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 600 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:34:18.739844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c336cd-3c84-4737-bdb5-0421cc03e229 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:18.740540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c336cd-3c84-4737-bdb5-0421cc03e229 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:18.740540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c336cd-3c84-4737-bdb5-0421cc03e229 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:18.741940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 702/1404] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:34:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:18.747245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61193fe1-c7a6-4199-83db-71559ad6fa29 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:18.747801 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61193fe1-c7a6-4199-83db-71559ad6fa29 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:18.747893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61193fe1-c7a6-4199-83db-71559ad6fa29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:18.822626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-61193fe1-c7a6-4199-83db-71559ad6fa29 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:18.823591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-61193fe1-c7a6-4199-83db-71559ad6fa29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:18.953273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:18.970174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 703/1405] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:34:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:19.338761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c336cd-3c84-4737-bdb5-0421cc03e229 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:19.340300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-84c336cd-3c84-4737-bdb5-0421cc03e229 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:19.487120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:19.563096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c336cd-3c84-4737-bdb5-0421cc03e229 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:19.715644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1386f4da-9dc6-4a23-b62e-047dad1c2d42 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:19.717704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1386f4da-9dc6-4a23-b62e-047dad1c2d42 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:19.718146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1386f4da-9dc6-4a23-b62e-047dad1c2d42 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:19.718536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1386f4da-9dc6-4a23-b62e-047dad1c2d42 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:19.720645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1386f4da-9dc6-4a23-b62e-047dad1c2d42 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:19.804551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:19.825256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1386f4da-9dc6-4a23-b62e-047dad1c2d42 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:19.827630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1386f4da-9dc6-4a23-b62e-047dad1c2d42 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:19.846318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 703/1406] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:21.974018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13ec2b-9b36-40a5-9941-14a18560d0a4 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:21.974609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13ec2b-9b36-40a5-9941-14a18560d0a4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:21.974609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13ec2b-9b36-40a5-9941-14a18560d0a4 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:21.976112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 704/1407] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:34:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:21.981368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429624f-4a45-4bc5-9cba-17419d47afec None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:21.981888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429624f-4a45-4bc5-9cba-17419d47afec None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:21.982005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429624f-4a45-4bc5-9cba-17419d47afec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:22.057945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2429624f-4a45-4bc5-9cba-17419d47afec None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:22.059601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2429624f-4a45-4bc5-9cba-17419d47afec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:22.205881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 704/1408] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:34:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:22.502431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a13ec2b-9b36-40a5-9941-14a18560d0a4 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:22.504179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4a13ec2b-9b36-40a5-9941-14a18560d0a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:22.716150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a13ec2b-9b36-40a5-9941-14a18560d0a4 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:22.824904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:22.886502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1326ce-3f16-4c02-8f31-1f72b9c8dee4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:22.887584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1326ce-3f16-4c02-8f31-1f72b9c8dee4 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:22.888072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1326ce-3f16-4c02-8f31-1f72b9c8dee4 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:22.888682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1326ce-3f16-4c02-8f31-1f72b9c8dee4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:22.890785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2f1326ce-3f16-4c02-8f31-1f72b9c8dee4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:23.021297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1326ce-3f16-4c02-8f31-1f72b9c8dee4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:23.022963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1326ce-3f16-4c02-8f31-1f72b9c8dee4 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:23.049167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 705/1409] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 547 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:32.203775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=79 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:33.040947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=64 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:38.551000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e6de6-fa56-4231-b4c6-c47af1917216 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:38.551566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e6de6-fa56-4231-b4c6-c47af1917216 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:38.551566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e6de6-fa56-4231-b4c6-c47af1917216 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:38.552904 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 705/1410] 10.222.0.77 () {54 vars in 918 bytes} [Mon Aug 29 22:34:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:38.558593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ff0e6f-90bf-4599-8df4-ed848473659c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:38.559136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ff0e6f-90bf-4599-8df4-ed848473659c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:38.559136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ff0e6f-90bf-4599-8df4-ed848473659c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:38.643742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-56ff0e6f-90bf-4599-8df4-ed848473659c None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:38.644984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-56ff0e6f-90bf-4599-8df4-ed848473659c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:38.765207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 706/1411] 10.222.0.77 () {58 vars in 1015 bytes} [Mon Aug 29 22:34:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:38.782528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442012c-9ad1-4860-a81d-f9a5d9306a27 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:38.782853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442012c-9ad1-4860-a81d-f9a5d9306a27 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:38.783141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442012c-9ad1-4860-a81d-f9a5d9306a27 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:38.784474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 706/1412] 10.222.0.77 () {54 vars in 918 bytes} [Mon Aug 29 22:34:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:38.789669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fcb0b-fe44-4052-ad1d-c81c46e1843e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:38.789959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fcb0b-fe44-4052-ad1d-c81c46e1843e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:38.790241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fcb0b-fe44-4052-ad1d-c81c46e1843e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:38.879346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:38.883094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-627fcb0b-fe44-4052-ad1d-c81c46e1843e None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:38.884896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-627fcb0b-fe44-4052-ad1d-c81c46e1843e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:39.009231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 707/1413] 10.222.0.77 () {58 vars in 1015 bytes} [Mon Aug 29 22:34:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:41.501444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a2e5d1-64d1-46df-acab-d75fd728187b None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:41.502058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a2e5d1-64d1-46df-acab-d75fd728187b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:41.502058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a2e5d1-64d1-46df-acab-d75fd728187b None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:41.503366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 707/1414] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:34:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:34:41.509249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47729934-acb0-433e-aa88-239888412a79 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:41.509674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47729934-acb0-433e-aa88-239888412a79 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:41.509784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47729934-acb0-433e-aa88-239888412a79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:41.587013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-47729934-acb0-433e-aa88-239888412a79 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:41.588290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-47729934-acb0-433e-aa88-239888412a79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:41.655534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:41.701633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 708/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:34:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:42.010596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a2e5d1-64d1-46df-acab-d75fd728187b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:42.013133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e9a2e5d1-64d1-46df-acab-d75fd728187b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:42.189373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a2e5d1-64d1-46df-acab-d75fd728187b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:42.269260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:42.317189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744f12a4-6789-4f4c-a4aa-ca9ca03f881f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:42.317802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f12a4-6789-4f4c-a4aa-ca9ca03f881f service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:42.318088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f12a4-6789-4f4c-a4aa-ca9ca03f881f service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:42.318361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f12a4-6789-4f4c-a4aa-ca9ca03f881f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:42.320232 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-744f12a4-6789-4f4c-a4aa-ca9ca03f881f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:42.421465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744f12a4-6789-4f4c-a4aa-ca9ca03f881f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:42.423117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744f12a4-6789-4f4c-a4aa-ca9ca03f881f service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:42.443424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 708/1416] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:43.381443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4af86-9305-4582-8e5d-9d9a3409a09e None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:43.382033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4af86-9305-4582-8e5d-9d9a3409a09e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:43.382033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a4af86-9305-4582-8e5d-9d9a3409a09e None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:43.383387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 709/1417] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:34:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:43.389134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11be37-ace7-4e81-88cb-518e82edd77a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:43.390321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11be37-ace7-4e81-88cb-518e82edd77a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:43.390321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11be37-ace7-4e81-88cb-518e82edd77a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:43.473270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-eb11be37-ace7-4e81-88cb-518e82edd77a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:43.474512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-eb11be37-ace7-4e81-88cb-518e82edd77a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:43.504749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:43.580295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 709/1418] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:34:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:43.735423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a4af86-9305-4582-8e5d-9d9a3409a09e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:43.737009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-18a4af86-9305-4582-8e5d-9d9a3409a09e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:43.911830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a4af86-9305-4582-8e5d-9d9a3409a09e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:43.955174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:44.035301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fe7a4f-6953-447b-819e-d9ba0dd3aa9a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:44.035837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fe7a4f-6953-447b-819e-d9ba0dd3aa9a service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:44.036115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fe7a4f-6953-447b-819e-d9ba0dd3aa9a service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:44.036387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fe7a4f-6953-447b-819e-d9ba0dd3aa9a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:44.038029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-01fe7a4f-6953-447b-819e-d9ba0dd3aa9a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:44.140932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fe7a4f-6953-447b-819e-d9ba0dd3aa9a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:44.142552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fe7a4f-6953-447b-819e-d9ba0dd3aa9a service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:44.162857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 710/1419] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:45.070055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836dd249-abb3-4575-a713-ad5b488f07a1 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:45.070658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836dd249-abb3-4575-a713-ad5b488f07a1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:45.070658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836dd249-abb3-4575-a713-ad5b488f07a1 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:45.071898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 710/1420] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:34:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:45.077854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89356e1-281d-4acc-af0b-d07d665f1f96 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:45.078393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89356e1-281d-4acc-af0b-d07d665f1f96 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:45.078393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89356e1-281d-4acc-af0b-d07d665f1f96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:45.140535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:45.164115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c89356e1-281d-4acc-af0b-d07d665f1f96 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:45.165350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c89356e1-281d-4acc-af0b-d07d665f1f96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:45.270751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 711/1421] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:34:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:45.426373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836dd249-abb3-4575-a713-ad5b488f07a1 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:45.427905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-836dd249-abb3-4575-a713-ad5b488f07a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:45.584187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836dd249-abb3-4575-a713-ad5b488f07a1 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:45.597710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:45.720700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be241717-ca93-406d-bfae-70ac90d98a9b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:45.721477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be241717-ca93-406d-bfae-70ac90d98a9b service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:45.721749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be241717-ca93-406d-bfae-70ac90d98a9b service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:45.722020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be241717-ca93-406d-bfae-70ac90d98a9b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:45.723721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-be241717-ca93-406d-bfae-70ac90d98a9b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:45.819678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be241717-ca93-406d-bfae-70ac90d98a9b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:45.821582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be241717-ca93-406d-bfae-70ac90d98a9b service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:45.841204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 711/1422] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:34:46.724073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e144fd8a-40bd-4f5a-839d-cc2fd70a8fb1 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:46.724664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e144fd8a-40bd-4f5a-839d-cc2fd70a8fb1 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:46.724664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e144fd8a-40bd-4f5a-839d-cc2fd70a8fb1 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:46.726042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 712/1423] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:34:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:46.732687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6cab19-0c9e-4aba-98d9-2be02ac14722 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:46.733244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6cab19-0c9e-4aba-98d9-2be02ac14722 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:46.733348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6cab19-0c9e-4aba-98d9-2be02ac14722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:46.755265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:46.832866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7e6cab19-0c9e-4aba-98d9-2be02ac14722 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:46.834237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7e6cab19-0c9e-4aba-98d9-2be02ac14722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:46.959963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 712/1424] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:47.113514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e144fd8a-40bd-4f5a-839d-cc2fd70a8fb1 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:47.115105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e144fd8a-40bd-4f5a-839d-cc2fd70a8fb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:47.252381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:47.292641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e144fd8a-40bd-4f5a-839d-cc2fd70a8fb1 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:47.422947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a2aac3-f128-48d0-b7e0-e44cb5077bf4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:47.423842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a2aac3-f128-48d0-b7e0-e44cb5077bf4 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:47.424124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a2aac3-f128-48d0-b7e0-e44cb5077bf4 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:47.424810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a2aac3-f128-48d0-b7e0-e44cb5077bf4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:47.427015 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a1a2aac3-f128-48d0-b7e0-e44cb5077bf4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:47.516194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:47.530999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a2aac3-f128-48d0-b7e0-e44cb5077bf4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:47.532633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a2aac3-f128-48d0-b7e0-e44cb5077bf4 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:47.558050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 713/1425] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:48.470792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3786fb2c-a584-41e1-9b4a-41db48dc5296 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:48.471353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3786fb2c-a584-41e1-9b4a-41db48dc5296 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:48.471353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3786fb2c-a584-41e1-9b4a-41db48dc5296 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:48.472651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 713/1426] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:34:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:48.477850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309040c5-a862-442e-b6a8-9108805d811a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:48.478391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309040c5-a862-442e-b6a8-9108805d811a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:48.479141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309040c5-a862-442e-b6a8-9108805d811a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:48.563667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-309040c5-a862-442e-b6a8-9108805d811a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:48.564919 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-309040c5-a862-442e-b6a8-9108805d811a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:48.674304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 714/1427] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:34:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:48.833329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3786fb2c-a584-41e1-9b4a-41db48dc5296 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:48.835171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3786fb2c-a584-41e1-9b4a-41db48dc5296 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:48.932279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:48.999835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3786fb2c-a584-41e1-9b4a-41db48dc5296 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:49.141480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a071402e-5093-4210-bf1c-b159c345a14b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:49.142004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071402e-5093-4210-bf1c-b159c345a14b service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:49.142275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071402e-5093-4210-bf1c-b159c345a14b service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:49.142551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071402e-5093-4210-bf1c-b159c345a14b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:49.144401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a071402e-5093-4210-bf1c-b159c345a14b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:49.199639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:49.253432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a071402e-5093-4210-bf1c-b159c345a14b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:49.255898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a071402e-5093-4210-bf1c-b159c345a14b service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:49.277059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 714/1428] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:50.099215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b068bad-c6b6-4320-a240-9583257c7603 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:50.099752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b068bad-c6b6-4320-a240-9583257c7603 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:50.099752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b068bad-c6b6-4320-a240-9583257c7603 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:50.100979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 715/1429] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:34:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:50.107412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff342228-d50a-4a6d-95ac-4f0e58cd20ad None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:50.107898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff342228-d50a-4a6d-95ac-4f0e58cd20ad None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:50.108017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff342228-d50a-4a6d-95ac-4f0e58cd20ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:50.192849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-ff342228-d50a-4a6d-95ac-4f0e58cd20ad None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:50.194108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ff342228-d50a-4a6d-95ac-4f0e58cd20ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:50.301456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 715/1430] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:34:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:50.452084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b068bad-c6b6-4320-a240-9583257c7603 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:50.453560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7b068bad-c6b6-4320-a240-9583257c7603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:50.508846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:50.625070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b068bad-c6b6-4320-a240-9583257c7603 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:50.788023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae963cf-8314-4d1c-91ca-ae99f7e9e888 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:50.788636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae963cf-8314-4d1c-91ca-ae99f7e9e888 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:50.788913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae963cf-8314-4d1c-91ca-ae99f7e9e888 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:50.789179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae963cf-8314-4d1c-91ca-ae99f7e9e888 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:50.790856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4ae963cf-8314-4d1c-91ca-ae99f7e9e888 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:50.800159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:50.887904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae963cf-8314-4d1c-91ca-ae99f7e9e888 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:50.889641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae963cf-8314-4d1c-91ca-ae99f7e9e888 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:50.908673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 716/1431] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:51.732492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ac0b78-c54b-4906-b1df-7ddbc1802fe7 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:51.733092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ac0b78-c54b-4906-b1df-7ddbc1802fe7 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:51.733092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ac0b78-c54b-4906-b1df-7ddbc1802fe7 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:51.734433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 716/1432] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:34:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:51.743331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75080f-b656-466a-872c-44c599dedf30 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:51.746273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75080f-b656-466a-872c-44c599dedf30 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:51.746273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75080f-b656-466a-872c-44c599dedf30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:51.833579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-6d75080f-b656-466a-872c-44c599dedf30 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:51.834869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6d75080f-b656-466a-872c-44c599dedf30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:51.917139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:51.956107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 717/1433] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:34:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:52.112829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ac0b78-c54b-4906-b1df-7ddbc1802fe7 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:52.114273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-40ac0b78-c54b-4906-b1df-7ddbc1802fe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:52.136628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:52.284618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ac0b78-c54b-4906-b1df-7ddbc1802fe7 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:52.401771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:52.431783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fbc98d-634d-49ec-a995-5d09110fe070 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:52.432401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fbc98d-634d-49ec-a995-5d09110fe070 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:52.432731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fbc98d-634d-49ec-a995-5d09110fe070 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:52.432997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fbc98d-634d-49ec-a995-5d09110fe070 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:52.434829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-78fbc98d-634d-49ec-a995-5d09110fe070 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:52.542172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fbc98d-634d-49ec-a995-5d09110fe070 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:52.543977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fbc98d-634d-49ec-a995-5d09110fe070 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:52.563784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 717/1434] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:53.407109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b68e3-7740-4e83-9988-ab900d6861c3 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:53.407697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b68e3-7740-4e83-9988-ab900d6861c3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:53.407697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b68e3-7740-4e83-9988-ab900d6861c3 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:53.409043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 718/1435] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:34:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:34:53.414080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fde5b0-cbb4-4c8d-95a0-0c6d8eb666ee None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:53.414543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fde5b0-cbb4-4c8d-95a0-0c6d8eb666ee None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:53.414650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fde5b0-cbb4-4c8d-95a0-0c6d8eb666ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:53.491746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a1fde5b0-cbb4-4c8d-95a0-0c6d8eb666ee None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:53.492968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a1fde5b0-cbb4-4c8d-95a0-0c6d8eb666ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:53.535216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:53.591937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 718/1436] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:34:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:53.739125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569b68e3-7740-4e83-9988-ab900d6861c3 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:53.740668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-569b68e3-7740-4e83-9988-ab900d6861c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:53.898898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569b68e3-7740-4e83-9988-ab900d6861c3 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:53.961675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:54.026655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66565fc9-d69e-4ccf-8cdc-1792f53e7dd3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:54.027214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66565fc9-d69e-4ccf-8cdc-1792f53e7dd3 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:54.027494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66565fc9-d69e-4ccf-8cdc-1792f53e7dd3 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:54.027769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66565fc9-d69e-4ccf-8cdc-1792f53e7dd3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:54.029446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-66565fc9-d69e-4ccf-8cdc-1792f53e7dd3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:54.123646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66565fc9-d69e-4ccf-8cdc-1792f53e7dd3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:54.125261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66565fc9-d69e-4ccf-8cdc-1792f53e7dd3 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:54.143660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 719/1437] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:54.959465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7727bd06-60cd-4c26-afc4-a0991d167294 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:54.960013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7727bd06-60cd-4c26-afc4-a0991d167294 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:54.960105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7727bd06-60cd-4c26-afc4-a0991d167294 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:54.961396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 719/1438] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:34:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:54.966874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3941f79f-1983-42f5-8582-2bda4fac567a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:54.967432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3941f79f-1983-42f5-8582-2bda4fac567a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:54.967432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3941f79f-1983-42f5-8582-2bda4fac567a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:55.045993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3941f79f-1983-42f5-8582-2bda4fac567a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:55.047263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3941f79f-1983-42f5-8582-2bda4fac567a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:55.049377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:55.149934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 720/1439] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:34:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:34:55.300194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7727bd06-60cd-4c26-afc4-a0991d167294 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:55.301721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7727bd06-60cd-4c26-afc4-a0991d167294 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:55.450932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7727bd06-60cd-4c26-afc4-a0991d167294 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:55.477151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:55.566554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b39fde-803d-45ca-b0e2-50b1a3f396f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:55.567184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b39fde-803d-45ca-b0e2-50b1a3f396f3 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:55.567462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b39fde-803d-45ca-b0e2-50b1a3f396f3 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:55.567750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b39fde-803d-45ca-b0e2-50b1a3f396f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:55.569348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-46b39fde-803d-45ca-b0e2-50b1a3f396f3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:55.661434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b39fde-803d-45ca-b0e2-50b1a3f396f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:55.663089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b39fde-803d-45ca-b0e2-50b1a3f396f3 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:55.683659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 720/1440] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:34:57.526562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c448d-4b6f-4c22-a6b0-9bec5fc2985e None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:57.527336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c448d-4b6f-4c22-a6b0-9bec5fc2985e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:57.527336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c448d-4b6f-4c22-a6b0-9bec5fc2985e None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:57.528513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 721/1441] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:34:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:34:57.534106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e40aecb-7649-4029-9086-deb6d57e48a3 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:57.534574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e40aecb-7649-4029-9086-deb6d57e48a3 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:57.534681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e40aecb-7649-4029-9086-deb6d57e48a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:57.578622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:57.606184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3e40aecb-7649-4029-9086-deb6d57e48a3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:34:57.607448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3e40aecb-7649-4029-9086-deb6d57e48a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:57.704827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 721/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:34:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:34:57.991936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4c448d-4b6f-4c22-a6b0-9bec5fc2985e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:34:57.994358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-bb4c448d-4b6f-4c22-a6b0-9bec5fc2985e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:58.099058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4c448d-4b6f-4c22-a6b0-9bec5fc2985e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:34:58.132662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:34:58.174753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d293ae-db85-48a5-88f6-4d90520533f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:34:58.175380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d293ae-db85-48a5-88f6-4d90520533f2 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:34:58.175664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d293ae-db85-48a5-88f6-4d90520533f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:34:58.175934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d293ae-db85-48a5-88f6-4d90520533f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:34:58.177555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-74d293ae-db85-48a5-88f6-4d90520533f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:34:58.286711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d293ae-db85-48a5-88f6-4d90520533f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:34:58.288190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d293ae-db85-48a5-88f6-4d90520533f2 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:34:58.307217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 722/1443] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:34:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:00.453817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9811dac-31ef-4b30-9b28-640da02c6b86 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:00.454419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9811dac-31ef-4b30-9b28-640da02c6b86 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:00.454419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9811dac-31ef-4b30-9b28-640da02c6b86 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:00.455742 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 722/1444] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:00.465028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f8881a-93c9-499e-97d9-83b995643b1b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:00.465510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f8881a-93c9-499e-97d9-83b995643b1b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:00.465628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f8881a-93c9-499e-97d9-83b995643b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:00.542424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-88f8881a-93c9-499e-97d9-83b995643b1b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:00.543709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-88f8881a-93c9-499e-97d9-83b995643b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:00.557512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:00.647373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 723/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:00.929042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9811dac-31ef-4b30-9b28-640da02c6b86 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:00.930604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a9811dac-31ef-4b30-9b28-640da02c6b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:01.042473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9811dac-31ef-4b30-9b28-640da02c6b86 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:01.056020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:01.122284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fecff20-0809-4e95-bbae-7c269553312e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:01.122864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fecff20-0809-4e95-bbae-7c269553312e service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:01.123143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fecff20-0809-4e95-bbae-7c269553312e service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:01.123418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fecff20-0809-4e95-bbae-7c269553312e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:01.125076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4fecff20-0809-4e95-bbae-7c269553312e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:01.252519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fecff20-0809-4e95-bbae-7c269553312e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:01.253999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fecff20-0809-4e95-bbae-7c269553312e service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:01.273660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 723/1446] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:03.539239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b334f825-f127-4c91-acee-a5a05837dd58 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:03.539819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b334f825-f127-4c91-acee-a5a05837dd58 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:03.539819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b334f825-f127-4c91-acee-a5a05837dd58 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:03.541191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 724/1447] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:03.546395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b4e52-9797-4ec2-b109-78b91a99943f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:03.546818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b4e52-9797-4ec2-b109-78b91a99943f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:03.546976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b4e52-9797-4ec2-b109-78b91a99943f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:03.595474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:03.613447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-819b4e52-9797-4ec2-b109-78b91a99943f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:03.614660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-819b4e52-9797-4ec2-b109-78b91a99943f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:03.712787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 724/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:04.000752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b334f825-f127-4c91-acee-a5a05837dd58 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:04.002350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b334f825-f127-4c91-acee-a5a05837dd58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:04.108192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b334f825-f127-4c91-acee-a5a05837dd58 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:04.149252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:04.184745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad72d99-61b0-4057-affb-27e38341ba39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:04.185352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad72d99-61b0-4057-affb-27e38341ba39 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:04.185629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad72d99-61b0-4057-affb-27e38341ba39 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:04.185899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad72d99-61b0-4057-affb-27e38341ba39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:04.187542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3ad72d99-61b0-4057-affb-27e38341ba39 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:04.295093 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad72d99-61b0-4057-affb-27e38341ba39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:04.296525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad72d99-61b0-4057-affb-27e38341ba39 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:04.314662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 725/1449] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:35:06.353106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c593f6-bf47-4c0f-9acf-13f36c9da222 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:06.353678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c593f6-bf47-4c0f-9acf-13f36c9da222 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:06.353678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c593f6-bf47-4c0f-9acf-13f36c9da222 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:06.355310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 725/1450] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:06.360442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49574b79-5e0b-47d5-a059-84675a933864 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:06.360876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49574b79-5e0b-47d5-a059-84675a933864 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:06.360974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49574b79-5e0b-47d5-a059-84675a933864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:06.427527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-49574b79-5e0b-47d5-a059-84675a933864 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:06.428734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-49574b79-5e0b-47d5-a059-84675a933864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:06.450401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:06.527459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 726/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:06.814993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c593f6-bf47-4c0f-9acf-13f36c9da222 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:06.818534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-66c593f6-bf47-4c0f-9acf-13f36c9da222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:06.932115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c593f6-bf47-4c0f-9acf-13f36c9da222 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:06.951932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:07.007259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92450b71-68d4-42b7-bb0b-659eb840d457 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:07.008038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92450b71-68d4-42b7-bb0b-659eb840d457 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:07.008268 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92450b71-68d4-42b7-bb0b-659eb840d457 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:07.008545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92450b71-68d4-42b7-bb0b-659eb840d457 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:07.010157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-92450b71-68d4-42b7-bb0b-659eb840d457 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:07.119726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92450b71-68d4-42b7-bb0b-659eb840d457 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:07.121513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92450b71-68d4-42b7-bb0b-659eb840d457 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:07.140573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 726/1452] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:09.114472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d5352-7b11-43ce-a738-76c8f8336a67 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:09.115077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d5352-7b11-43ce-a738-76c8f8336a67 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:09.115077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d5352-7b11-43ce-a738-76c8f8336a67 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:09.116411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 727/1453] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:09.122232 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121f592-7af7-4eec-a221-2238e8553293 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:09.122675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121f592-7af7-4eec-a221-2238e8553293 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:09.122904 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121f592-7af7-4eec-a221-2238e8553293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:09.187580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:09.190486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-4121f592-7af7-4eec-a221-2238e8553293 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:09.192417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4121f592-7af7-4eec-a221-2238e8553293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:09.290309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 727/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:09.571558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33d5352-7b11-43ce-a738-76c8f8336a67 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:09.573160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c33d5352-7b11-43ce-a738-76c8f8336a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:09.678227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33d5352-7b11-43ce-a738-76c8f8336a67 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:09.728998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:09.754681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9ae5a7-b111-4fa2-a0b0-d2c25c57cbf7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:09.755323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ae5a7-b111-4fa2-a0b0-d2c25c57cbf7 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:09.755601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ae5a7-b111-4fa2-a0b0-d2c25c57cbf7 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:09.755871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ae5a7-b111-4fa2-a0b0-d2c25c57cbf7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:09.758137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-eb9ae5a7-b111-4fa2-a0b0-d2c25c57cbf7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:09.999555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9ae5a7-b111-4fa2-a0b0-d2c25c57cbf7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:10.001000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9ae5a7-b111-4fa2-a0b0-d2c25c57cbf7 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:10.020067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 728/1455] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:12.037839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d8e61d-c132-43da-88f7-50a70c628e69 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:12.038995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d8e61d-c132-43da-88f7-50a70c628e69 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:12.038995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d8e61d-c132-43da-88f7-50a70c628e69 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:12.040266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 728/1456] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:12.045908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf9f4b7-df78-4934-ae89-58c456d56d65 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:12.046381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf9f4b7-df78-4934-ae89-58c456d56d65 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:12.046497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf9f4b7-df78-4934-ae89-58c456d56d65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:12.112503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-adf9f4b7-df78-4934-ae89-58c456d56d65 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:12.113625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-adf9f4b7-df78-4934-ae89-58c456d56d65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:12.143179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:12.220955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 729/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:12.516574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d8e61d-c132-43da-88f7-50a70c628e69 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:12.518166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-82d8e61d-c132-43da-88f7-50a70c628e69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:12.636190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d8e61d-c132-43da-88f7-50a70c628e69 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:12.673761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:12.725779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f887f094-ae44-4767-a7c3-54b1980ad611 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:12.726388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f887f094-ae44-4767-a7c3-54b1980ad611 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:12.726668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f887f094-ae44-4767-a7c3-54b1980ad611 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:12.726966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f887f094-ae44-4767-a7c3-54b1980ad611 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:12.728586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f887f094-ae44-4767-a7c3-54b1980ad611 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:12.849190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f887f094-ae44-4767-a7c3-54b1980ad611 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:12.850651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f887f094-ae44-4767-a7c3-54b1980ad611 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:12.868756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 729/1458] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:14.851196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab38cda4-9bdd-47bd-8394-3cca1fb496b0 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:14.851733 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab38cda4-9bdd-47bd-8394-3cca1fb496b0 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:14.851847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab38cda4-9bdd-47bd-8394-3cca1fb496b0 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:14.853170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 730/1459] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:14.859194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6595190-e079-482b-8fa5-c84275412154 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:14.859661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6595190-e079-482b-8fa5-c84275412154 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:14.859759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6595190-e079-482b-8fa5-c84275412154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:14.926465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e6595190-e079-482b-8fa5-c84275412154 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:14.927736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e6595190-e079-482b-8fa5-c84275412154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:14.936032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:15.026389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 730/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:15.315934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab38cda4-9bdd-47bd-8394-3cca1fb496b0 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:15.317528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ab38cda4-9bdd-47bd-8394-3cca1fb496b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:15.423072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab38cda4-9bdd-47bd-8394-3cca1fb496b0 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:15.479359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:15.497983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b62b85-3296-4051-8cea-669d679eebf9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:15.498591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b62b85-3296-4051-8cea-669d679eebf9 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:15.498904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b62b85-3296-4051-8cea-669d679eebf9 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:15.499180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b62b85-3296-4051-8cea-669d679eebf9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:15.500799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f0b62b85-3296-4051-8cea-669d679eebf9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:15.610147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b62b85-3296-4051-8cea-669d679eebf9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:15.611533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b62b85-3296-4051-8cea-669d679eebf9 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:15.630006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 731/1461] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:17.619453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eecdad1-13ec-424a-a7d9-69b03d89e9f2 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:17.619909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eecdad1-13ec-424a-a7d9-69b03d89e9f2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:17.620013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eecdad1-13ec-424a-a7d9-69b03d89e9f2 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:17.621311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 731/1462] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:17.627031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ae2e6b-ff73-49cc-b85f-764f41b21a8e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:17.627461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ae2e6b-ff73-49cc-b85f-764f41b21a8e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:17.627554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ae2e6b-ff73-49cc-b85f-764f41b21a8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:17.694564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-41ae2e6b-ff73-49cc-b85f-764f41b21a8e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:17.695817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-41ae2e6b-ff73-49cc-b85f-764f41b21a8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:17.733400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:17.792388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 732/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:35:18.080165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eecdad1-13ec-424a-a7d9-69b03d89e9f2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:18.081763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4eecdad1-13ec-424a-a7d9-69b03d89e9f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:18.186649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eecdad1-13ec-424a-a7d9-69b03d89e9f2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:18.224564 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:18.269456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8d28c7-2164-40e8-81eb-081dd57eef79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:18.270069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8d28c7-2164-40e8-81eb-081dd57eef79 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:18.270353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8d28c7-2164-40e8-81eb-081dd57eef79 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:18.270637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8d28c7-2164-40e8-81eb-081dd57eef79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:18.272808 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ec8d28c7-2164-40e8-81eb-081dd57eef79 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:18.383094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8d28c7-2164-40e8-81eb-081dd57eef79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:18.384454 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8d28c7-2164-40e8-81eb-081dd57eef79 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:18.403101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 732/1464] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:20.344483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974eaa64-63b3-40f5-8c85-6fe594bd001d None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:20.345049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974eaa64-63b3-40f5-8c85-6fe594bd001d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:20.345049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974eaa64-63b3-40f5-8c85-6fe594bd001d None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:20.346396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 733/1465] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:20.351764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18318adb-f780-4ed5-bb98-a68b474dddc3 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:20.352203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18318adb-f780-4ed5-bb98-a68b474dddc3 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:20.352348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18318adb-f780-4ed5-bb98-a68b474dddc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:20.420555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-18318adb-f780-4ed5-bb98-a68b474dddc3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:20.421680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-18318adb-f780-4ed5-bb98-a68b474dddc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:20.439357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:20.517422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 733/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:35:20.797409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974eaa64-63b3-40f5-8c85-6fe594bd001d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:20.799010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-974eaa64-63b3-40f5-8c85-6fe594bd001d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:20.901669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974eaa64-63b3-40f5-8c85-6fe594bd001d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:20.968245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:20.976019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8ebe8-dd0d-4fcc-aac5-50862bb214e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:20.976623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8ebe8-dd0d-4fcc-aac5-50862bb214e4 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:20.976903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8ebe8-dd0d-4fcc-aac5-50862bb214e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:20.977310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8ebe8-dd0d-4fcc-aac5-50862bb214e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:20.979227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d5f8ebe8-dd0d-4fcc-aac5-50862bb214e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:21.088890 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f8ebe8-dd0d-4fcc-aac5-50862bb214e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:21.090252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f8ebe8-dd0d-4fcc-aac5-50862bb214e4 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:21.108560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 734/1467] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:23.040457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6137a-d055-4ef1-89f1-6889e5969b4f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:23.040966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6137a-d055-4ef1-89f1-6889e5969b4f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:23.041099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6137a-d055-4ef1-89f1-6889e5969b4f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:23.042350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 734/1468] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:35:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:23.047568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5555910-0864-4f6b-9c4a-649ab16e370a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:23.047999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5555910-0864-4f6b-9c4a-649ab16e370a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:23.048097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5555910-0864-4f6b-9c4a-649ab16e370a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:23.115836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d5555910-0864-4f6b-9c4a-649ab16e370a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:23.117016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d5555910-0864-4f6b-9c4a-649ab16e370a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:23.165199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:23.212650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 735/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:23.489211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be6137a-d055-4ef1-89f1-6889e5969b4f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:23.491434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6be6137a-d055-4ef1-89f1-6889e5969b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:23.592515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be6137a-d055-4ef1-89f1-6889e5969b4f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:23.639150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:23.666181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f219c1-6bc3-42ca-85e5-4ccf48da0a6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:23.666814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f219c1-6bc3-42ca-85e5-4ccf48da0a6b service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:23.667092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f219c1-6bc3-42ca-85e5-4ccf48da0a6b service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:23.667361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f219c1-6bc3-42ca-85e5-4ccf48da0a6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:23.668959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f7f219c1-6bc3-42ca-85e5-4ccf48da0a6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:23.900947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f219c1-6bc3-42ca-85e5-4ccf48da0a6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:23.902375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f219c1-6bc3-42ca-85e5-4ccf48da0a6b service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:23.920939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 735/1470] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:33.211246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=22 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:33.918667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=66 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:43.779652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82379c62-7987-4a53-860d-c335848f3c62 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:43.780195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82379c62-7987-4a53-860d-c335848f3c62 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:43.780324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82379c62-7987-4a53-860d-c335848f3c62 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:43.781737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 736/1471] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:35:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:43.789513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e45359-5c4b-48c2-bd8f-3738be536823 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:43.789997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e45359-5c4b-48c2-bd8f-3738be536823 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:43.790100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e45359-5c4b-48c2-bd8f-3738be536823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:43.858949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-69e45359-5c4b-48c2-bd8f-3738be536823 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:43.860234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-69e45359-5c4b-48c2-bd8f-3738be536823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:43.958074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 736/1472] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:35:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:44.260172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f688870-838c-4807-8209-bae89028bc5f None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:44.260451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f688870-838c-4807-8209-bae89028bc5f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:44.260719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f688870-838c-4807-8209-bae89028bc5f None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:44.262039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 737/1473] 10.222.0.77 () {54 vars in 967 bytes} [Mon Aug 29 22:35:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:44.267053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2417ee9f-e433-4209-865b-a52e018a9cc8 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:44.267346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2417ee9f-e433-4209-865b-a52e018a9cc8 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:44.267624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2417ee9f-e433-4209-865b-a52e018a9cc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:44.345561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2417ee9f-e433-4209-865b-a52e018a9cc8 None None] MFA Rules not processed for user `c1e61f3604fd4a23ae5f1e7e62ae21ae`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:44.346791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2417ee9f-e433-4209-865b-a52e018a9cc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:44.369161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:44.426512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 737/1474] 10.222.0.77 () {58 vars in 1064 bytes} [Mon Aug 29 22:35:44 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:44.434137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f688870-838c-4807-8209-bae89028bc5f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:44.435728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2f688870-838c-4807-8209-bae89028bc5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:44.541581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f688870-838c-4807-8209-bae89028bc5f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:44.618406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e10396b-7117-431d-9de3-58a7de2cd3cf service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:44.619072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10396b-7117-431d-9de3-58a7de2cd3cf service glance] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:44.619354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10396b-7117-431d-9de3-58a7de2cd3cf service glance] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:44.619630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10396b-7117-431d-9de3-58a7de2cd3cf service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:44.621293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0e10396b-7117-431d-9de3-58a7de2cd3cf service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:44.676817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:44.734555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e10396b-7117-431d-9de3-58a7de2cd3cf service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:44.736045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e10396b-7117-431d-9de3-58a7de2cd3cf service glance] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:44.757396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 738/1475] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:46.817297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9001c6-8dbd-4efd-a044-dc01e8a4b664 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:46.817902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9001c6-8dbd-4efd-a044-dc01e8a4b664 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:46.817902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9001c6-8dbd-4efd-a044-dc01e8a4b664 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:46.819596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 738/1476] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:35:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:46.826047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bfd254-c146-4fdf-987a-91fdb06d1613 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:46.826485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bfd254-c146-4fdf-987a-91fdb06d1613 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:46.826626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bfd254-c146-4fdf-987a-91fdb06d1613 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:46.898901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a2bfd254-c146-4fdf-987a-91fdb06d1613 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:46.900138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a2bfd254-c146-4fdf-987a-91fdb06d1613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:47.010905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 739/1477] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:35:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:47.292173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9001c6-8dbd-4efd-a044-dc01e8a4b664 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:47.293923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9f9001c6-8dbd-4efd-a044-dc01e8a4b664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:47.421980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9001c6-8dbd-4efd-a044-dc01e8a4b664 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:47.470584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:47.521012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c86730-c76a-448c-82f0-5fa1144d4a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:47.521783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c86730-c76a-448c-82f0-5fa1144d4a4a admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:47.521953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c86730-c76a-448c-82f0-5fa1144d4a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:47.522211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c86730-c76a-448c-82f0-5fa1144d4a4a admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:47.522919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c86730-c76a-448c-82f0-5fa1144d4a4a admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:47.524633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING py.warnings [None req-63c86730-c76a-448c-82f0-5fa1144d4a4a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:35:47.524633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: warnings.warn(msg) Aug 29 22:35:47.524633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: Aug 29 22:35:47.525749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c86730-c76a-448c-82f0-5fa1144d4a4a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:47.544747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 739/1478] 10.222.0.77 () {56 vars in 1108 bytes} [Mon Aug 29 22:35:47 2022] GET /identity/v3/projects => generated 2553 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:35:49.496817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a36ccc-b954-4fcf-956b-6690863e1ae9 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:49.497695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a36ccc-b954-4fcf-956b-6690863e1ae9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:49.497695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a36ccc-b954-4fcf-956b-6690863e1ae9 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:49.499324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 740/1479] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:35:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:49.504765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d81fa16-693e-4733-99d8-ec817836dde6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:49.506045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d81fa16-693e-4733-99d8-ec817836dde6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:49.506045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d81fa16-693e-4733-99d8-ec817836dde6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:49.577668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-8d81fa16-693e-4733-99d8-ec817836dde6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:49.578925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8d81fa16-693e-4733-99d8-ec817836dde6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:49.680970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 740/1480] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:35:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:49.975041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a36ccc-b954-4fcf-956b-6690863e1ae9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:49.976877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-85a36ccc-b954-4fcf-956b-6690863e1ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:49.987333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:50.127196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a36ccc-b954-4fcf-956b-6690863e1ae9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:50.211464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:50.244510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699d52d-c117-4b70-9101-bdab421c13c5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:50.245230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699d52d-c117-4b70-9101-bdab421c13c5 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:50.245599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699d52d-c117-4b70-9101-bdab421c13c5 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:50.245960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699d52d-c117-4b70-9101-bdab421c13c5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:50.247802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7699d52d-c117-4b70-9101-bdab421c13c5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:50.338092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7699d52d-c117-4b70-9101-bdab421c13c5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:50.339730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7699d52d-c117-4b70-9101-bdab421c13c5 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:50.359297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 741/1481] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:52.482080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d48a54-5836-4332-bee5-ed7b1a715eca None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:52.482672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d48a54-5836-4332-bee5-ed7b1a715eca None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:52.482672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d48a54-5836-4332-bee5-ed7b1a715eca None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:52.483992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 741/1482] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:35:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:52.489922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c078babf-3ab6-418a-90bf-278d27e4904c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:52.490516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c078babf-3ab6-418a-90bf-278d27e4904c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:52.490516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c078babf-3ab6-418a-90bf-278d27e4904c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:52.558031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c078babf-3ab6-418a-90bf-278d27e4904c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:52.559275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c078babf-3ab6-418a-90bf-278d27e4904c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:52.615384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:52.668309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 742/1483] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:35:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:52.961542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d48a54-5836-4332-bee5-ed7b1a715eca None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:52.963112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-60d48a54-5836-4332-bee5-ed7b1a715eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:52.987826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:53.071761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d48a54-5836-4332-bee5-ed7b1a715eca None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:53.149163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f16919-4357-47d4-b436-e299fed9d080 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:53.149789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f16919-4357-47d4-b436-e299fed9d080 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:53.150065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f16919-4357-47d4-b436-e299fed9d080 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:53.150335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f16919-4357-47d4-b436-e299fed9d080 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:53.151967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-84f16919-4357-47d4-b436-e299fed9d080 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:53.229582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:53.265524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f16919-4357-47d4-b436-e299fed9d080 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:53.266992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f16919-4357-47d4-b436-e299fed9d080 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:53.285938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 742/1484] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:35:55.291902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b05ffb5-bf5d-488e-8caf-e297b97a9d66 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:55.292459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b05ffb5-bf5d-488e-8caf-e297b97a9d66 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:55.292459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b05ffb5-bf5d-488e-8caf-e297b97a9d66 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:55.293883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 743/1485] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:35:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:55.299289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853da84-9808-49b2-94e3-9a3f5fd49993 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:55.299726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853da84-9808-49b2-94e3-9a3f5fd49993 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:55.300010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853da84-9808-49b2-94e3-9a3f5fd49993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:55.368456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9853da84-9808-49b2-94e3-9a3f5fd49993 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:55.369690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9853da84-9808-49b2-94e3-9a3f5fd49993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:55.482446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 743/1486] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:35:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:55.779940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b05ffb5-bf5d-488e-8caf-e297b97a9d66 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:55.781524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7b05ffb5-bf5d-488e-8caf-e297b97a9d66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:55.891497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b05ffb5-bf5d-488e-8caf-e297b97a9d66 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:55.961786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:55.963679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b48b87-988a-4ee2-b134-b252e8e9580d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:55.964341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b48b87-988a-4ee2-b134-b252e8e9580d service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:55.964682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b48b87-988a-4ee2-b134-b252e8e9580d service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:55.965144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b48b87-988a-4ee2-b134-b252e8e9580d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:55.967173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e9b48b87-988a-4ee2-b134-b252e8e9580d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:56.075896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b48b87-988a-4ee2-b134-b252e8e9580d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:56.077275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b48b87-988a-4ee2-b134-b252e8e9580d service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:56.095043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 744/1487] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:35:58.125969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf46695-717f-45c4-9570-ebe244caa536 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:58.126547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf46695-717f-45c4-9570-ebe244caa536 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:58.126547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf46695-717f-45c4-9570-ebe244caa536 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:58.127925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 744/1488] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:35:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:35:58.133281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2491d27f-49df-4c7a-8ef8-c19125d29cc0 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:58.133758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2491d27f-49df-4c7a-8ef8-c19125d29cc0 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:58.133865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2491d27f-49df-4c7a-8ef8-c19125d29cc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:58.200402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2491d27f-49df-4c7a-8ef8-c19125d29cc0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:35:58.201631 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2491d27f-49df-4c7a-8ef8-c19125d29cc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:58.255916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:58.301045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 745/1489] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:35:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:35:58.594311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf46695-717f-45c4-9570-ebe244caa536 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:35:58.595931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5cf46695-717f-45c4-9570-ebe244caa536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:58.634488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:58.707656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf46695-717f-45c4-9570-ebe244caa536 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:35:58.784906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0893a206-f133-4497-a52d-6f9cec86c73b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:35:58.785525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0893a206-f133-4497-a52d-6f9cec86c73b service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:35:58.785802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0893a206-f133-4497-a52d-6f9cec86c73b service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:35:58.786075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0893a206-f133-4497-a52d-6f9cec86c73b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:35:58.787776 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0893a206-f133-4497-a52d-6f9cec86c73b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:35:58.877895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:35:58.912047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0893a206-f133-4497-a52d-6f9cec86c73b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:35:58.913531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0893a206-f133-4497-a52d-6f9cec86c73b service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:35:58.938226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 745/1490] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:35:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:36:01.120617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fcaa9-b149-4ca2-8063-c221d8d2face None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:36:01.121165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fcaa9-b149-4ca2-8063-c221d8d2face None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:36:01.121165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fcaa9-b149-4ca2-8063-c221d8d2face None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:36:01.122457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 746/1491] 10.222.0.77 () {54 vars in 927 bytes} [Mon Aug 29 22:36:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:36:01.128373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2773b4f2-bfef-4b8f-bf50-9de1ea678cd7 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:36:01.128829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2773b4f2-bfef-4b8f-bf50-9de1ea678cd7 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:36:01.128937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2773b4f2-bfef-4b8f-bf50-9de1ea678cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:36:01.195875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2773b4f2-bfef-4b8f-bf50-9de1ea678cd7 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:36:01.197098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2773b4f2-bfef-4b8f-bf50-9de1ea678cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:36:01.296709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 746/1492] 10.222.0.77 () {58 vars in 1024 bytes} [Mon Aug 29 22:36:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:36:01.590939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6fcaa9-b149-4ca2-8063-c221d8d2face None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:36:01.592592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8e6fcaa9-b149-4ca2-8063-c221d8d2face None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:36:01.699830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6fcaa9-b149-4ca2-8063-c221d8d2face None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:36:01.775104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1af3c22-5b12-47f2-9db5-4eef18020643 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:36:01.775715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1af3c22-5b12-47f2-9db5-4eef18020643 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:36:01.775993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1af3c22-5b12-47f2-9db5-4eef18020643 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:36:01.776261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1af3c22-5b12-47f2-9db5-4eef18020643 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:36:01.777880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a1af3c22-5b12-47f2-9db5-4eef18020643 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:36:01.787313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:36:01.890462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1af3c22-5b12-47f2-9db5-4eef18020643 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:36:01.891871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1af3c22-5b12-47f2-9db5-4eef18020643 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:36:01.909728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 747/1493] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:36:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:36:11.295011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=82 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:36:11.907406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=50 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:15.274684 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31e6928-be0d-459a-8f6e-04a149d5d1d6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:15.275334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31e6928-be0d-459a-8f6e-04a149d5d1d6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:15.275334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31e6928-be0d-459a-8f6e-04a149d5d1d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:15.356791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e31e6928-be0d-459a-8f6e-04a149d5d1d6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:37:15.357985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e31e6928-be0d-459a-8f6e-04a149d5d1d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:15.459337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 747/1494] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:37:15.468177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2001ad-13cc-4611-8ea0-6d62ef8155c9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:15.468684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2001ad-13cc-4611-8ea0-6d62ef8155c9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:15.468794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2001ad-13cc-4611-8ea0-6d62ef8155c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:15.548352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-6f2001ad-13cc-4611-8ea0-6d62ef8155c9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:37:15.549562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6f2001ad-13cc-4611-8ea0-6d62ef8155c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:15.653774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 748/1495] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:37:15.658457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31e6928-be0d-459a-8f6e-04a149d5d1d6 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:15.659702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e31e6928-be0d-459a-8f6e-04a149d5d1d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:15.672414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31e6928-be0d-459a-8f6e-04a149d5d1d6 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:15.740460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:15.778259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:15.779001 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:15.779279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:15.779545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:15.780194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:37:15.780496 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:15.781516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING py.warnings [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:37:15.781516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: warnings.warn(msg) Aug 29 22:37:15.781516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: Aug 29 22:37:15.782398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:15.797643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 748/1496] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:37:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:37:15.805975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2001ad-13cc-4611-8ea0-6d62ef8155c9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:15.816181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2001ad-13cc-4611-8ea0-6d62ef8155c9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:15.880846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:15.915765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:15.916294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:15.916662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:15.916928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:15.917579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:15.919368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING py.warnings [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:37:15.919368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: warnings.warn(msg) Aug 29 22:37:15.919368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: Aug 29 22:37:15.921229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:15.982998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 749/1497] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:37:15 2022] POST /identity/v3/projects => generated 375 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:37:15.987254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:15.988771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:16.022987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406492e5-4442-44c1-8117-77c4192fc60b admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:16.118500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:16.119133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:16.119414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:16.119687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:16.120329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:16.121361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING py.warnings [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:37:16.121361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: warnings.warn(msg) Aug 29 22:37:16.121361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: Aug 29 22:37:16.122568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:16.167467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 749/1498] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:37:15 2022] POST /identity/v3/users => generated 288 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:37:16.172486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:16.174084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:16.237621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c627d719-7b3f-4b50-a37d-bc69907908c2 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:16.325660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:16.337882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:16.338495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:16.338802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:16.339076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:16.339723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:16.340742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING py.warnings [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:37:16.340742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: warnings.warn(msg) Aug 29 22:37:16.340742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: Aug 29 22:37:16.341638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:16.354905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 750/1499] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:37:16.359558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:16.361090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:16.373660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bfcd85-4587-4189-ae33-a2fa5218192e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:16.400389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:16.474337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03ad2b-e77c-40c0-832c-6f37859cb12f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:16.474984 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03ad2b-e77c-40c0-832c-6f37859cb12f admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:16.475263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03ad2b-e77c-40c0-832c-6f37859cb12f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:16.475533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03ad2b-e77c-40c0-832c-6f37859cb12f admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:16.476181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c03ad2b-e77c-40c0-832c-6f37859cb12f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:16.477206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING py.warnings [None req-2c03ad2b-e77c-40c0-832c-6f37859cb12f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:37:16.477206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: warnings.warn(msg) Aug 29 22:37:16.477206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: Aug 29 22:37:16.478095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c03ad2b-e77c-40c0-832c-6f37859cb12f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:16.491048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 750/1500] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:37:16.495264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:16.496811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:16.530199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1c12f9-95bc-4a90-a499-9cce2c37f027 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:16.628743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4b84cc-6b9d-4dae-96c7-0008bda38b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:16.629550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4b84cc-6b9d-4dae-96c7-0008bda38b4c admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:16.629834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4b84cc-6b9d-4dae-96c7-0008bda38b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:16.630101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4b84cc-6b9d-4dae-96c7-0008bda38b4c admin admin] PATH_INFO: `/v3/projects/c95a8aab73a54dd7b96d307d55197f9a/users/f781529ac17b4499bd3b426c56394dbe/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:16.652707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4b84cc-6b9d-4dae-96c7-0008bda38b4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c95a8aab73a54dd7b96d307d55197f9a, user_id=f781529ac17b4499bd3b426c56394dbe, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:16.654332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING py.warnings [None req-ec4b84cc-6b9d-4dae-96c7-0008bda38b4c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:37:16.654332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: warnings.warn(msg) Aug 29 22:37:16.654332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: Aug 29 22:37:16.656319 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4b84cc-6b9d-4dae-96c7-0008bda38b4c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:16.681349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 751/1501] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:37:16 2022] PUT /identity/v3/projects/c95a8aab73a54dd7b96d307d55197f9a/users/f781529ac17b4499bd3b426c56394dbe/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:37:16.693819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4dd00d-35c4-4f08-ae3f-d997613eee2b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:16.694103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4dd00d-35c4-4f08-ae3f-d997613eee2b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:16.694380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4dd00d-35c4-4f08-ae3f-d997613eee2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:16.752048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-ab4dd00d-35c4-4f08-ae3f-d997613eee2b None None] MFA Rules not processed for user `f781529ac17b4499bd3b426c56394dbe`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:37:16.753188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ab4dd00d-35c4-4f08-ae3f-d997613eee2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:16.833040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 751/1502] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:37:16 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:37:16.846506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41be1720-be72-46c4-a6b4-c6ea16799c13 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:16.846845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41be1720-be72-46c4-a6b4-c6ea16799c13 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:16.847127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41be1720-be72-46c4-a6b4-c6ea16799c13 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:16.848973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 752/1503] 10.222.0.77 () {54 vars in 964 bytes} [Mon Aug 29 22:37:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:37:16.853734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7c368-cd9a-43df-aa06-fb15c535f8b2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:16.854024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7c368-cd9a-43df-aa06-fb15c535f8b2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:16.854301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7c368-cd9a-43df-aa06-fb15c535f8b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:16.885267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:16.934032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-dfa7c368-cd9a-43df-aa06-fb15c535f8b2 None None] MFA Rules not processed for user `7964b4109f9a4ac6a68d90366489557c`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:37:16.935192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-dfa7c368-cd9a-43df-aa06-fb15c535f8b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:17.023320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 752/1504] 10.222.0.77 () {58 vars in 1061 bytes} [Mon Aug 29 22:37:16 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:37:17.033437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41be1720-be72-46c4-a6b4-c6ea16799c13 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:17.035080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-41be1720-be72-46c4-a6b4-c6ea16799c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:17.085276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:17.140337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41be1720-be72-46c4-a6b4-c6ea16799c13 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:17.214719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:17.215358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:17.215633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:17.215901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:17.217510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:17.318513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:17.320299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:17.323942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:17.339681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 753/1505] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:37:17 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:37:17.385374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4884d7d4-edb9-40a2-9162-ce1b71b79279 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:17.385817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4884d7d4-edb9-40a2-9162-ce1b71b79279 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:17.385925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4884d7d4-edb9-40a2-9162-ce1b71b79279 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:17.387162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 753/1506] 10.222.0.77 () {52 vars in 830 bytes} [Mon Aug 29 22:37:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:37:17.391530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:17.392741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:17.492960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d19fd6-30f8-4d9d-9a8d-2a004bb02569 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:17.586860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:17.587501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:17.587707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:17.587943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] PATH_INFO: `/v3/users/f781529ac17b4499bd3b426c56394dbe` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:17.600323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f781529ac17b4499bd3b426c56394dbe)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:17.601350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING py.warnings [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:37:17.601350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: warnings.warn(msg) Aug 29 22:37:17.601350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: Aug 29 22:37:17.602576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:17.612870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:17.672215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f781529ac17b4499bd3b426c56394dbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:17.687049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f781529ac17b4499bd3b426c56394dbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:17.687509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f781529ac17b4499bd3b426c56394dbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:17.688040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f781529ac17b4499bd3b426c56394dbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:17.696474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f781529ac17b4499bd3b426c56394dbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:17.725080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 754/1507] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:37:17 2022] DELETE /identity/v3/users/f781529ac17b4499bd3b426c56394dbe => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:37:17.729437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c76f67-cd3a-4fff-85ab-28be282ab78a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:17.729723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c76f67-cd3a-4fff-85ab-28be282ab78a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:17.730003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c76f67-cd3a-4fff-85ab-28be282ab78a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:17.787716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-29c76f67-cd3a-4fff-85ab-28be282ab78a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:37:17.788068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-29c76f67-cd3a-4fff-85ab-28be282ab78a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:17.864731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:17.878488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 754/1508] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:37:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:37:17.888159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:17.889704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:17.992196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22f71e-3867-43d2-b280-02207745979d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:18.046954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:18.068799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078e188a-1907-452a-b099-343ed80f1b9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:18.069412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e188a-1907-452a-b099-343ed80f1b9b service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:18.069694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e188a-1907-452a-b099-343ed80f1b9b service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:18.069969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e188a-1907-452a-b099-343ed80f1b9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:18.071628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-078e188a-1907-452a-b099-343ed80f1b9b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:18.183938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078e188a-1907-452a-b099-343ed80f1b9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:18.185809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078e188a-1907-452a-b099-343ed80f1b9b service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:18.204687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 755/1509] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:37:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:37:18.605264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c76f67-cd3a-4fff-85ab-28be282ab78a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:37:18.606554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-29c76f67-cd3a-4fff-85ab-28be282ab78a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:37:18.618872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c76f67-cd3a-4fff-85ab-28be282ab78a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:37:18.717646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:37:18.718195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:37:18.718474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:37:18.718771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] PATH_INFO: `/v3/projects/c95a8aab73a54dd7b96d307d55197f9a` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:37:18.728958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c95a8aab73a54dd7b96d307d55197f9a)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:37:18.729995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING py.warnings [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:37:18.729995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: warnings.warn(msg) Aug 29 22:37:18.729995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: Aug 29 22:37:18.731226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:37:18.789631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c95a8aab73a54dd7b96d307d55197f9a', 'request_id': 'req-dfc9fb36-a8cc-4e81-ae24-e7101b538377'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:18.805431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c95a8aab73a54dd7b96d307d55197f9a', 'request_id': 'req-dfc9fb36-a8cc-4e81-ae24-e7101b538377'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:18.815465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c95a8aab73a54dd7b96d307d55197f9a', 'request_id': 'req-dfc9fb36-a8cc-4e81-ae24-e7101b538377'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:18.838220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] The token cache is being invalidate because project c95a8aab73a54dd7b96d307d55197f9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:37:18.838506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:18.839069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dfc9fb36-a8cc-4e81-ae24-e7101b538377 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:37:18.840361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 755/1510] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:37:18 2022] DELETE /identity/v3/projects/c95a8aab73a54dd7b96d307d55197f9a => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:37:28.202317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=62 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:37:28.836206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:38:48.935207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a525c-dd98-403e-8645-249f3f35ccab None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:48.935805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a525c-dd98-403e-8645-249f3f35ccab None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:48.935805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a525c-dd98-403e-8645-249f3f35ccab None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:48.937143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 756/1511] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:38:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:38:48.942480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629fea6d-861e-48f6-beaa-130d14f051ae None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:48.942957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629fea6d-861e-48f6-beaa-130d14f051ae None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:48.943072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629fea6d-861e-48f6-beaa-130d14f051ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:49.026867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-629fea6d-861e-48f6-beaa-130d14f051ae None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:38:49.028109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-629fea6d-861e-48f6-beaa-130d14f051ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:38:49.140218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 756/1512] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:38:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:38:49.295638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a525c-dd98-403e-8645-249f3f35ccab None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:38:49.297824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ec5a525c-dd98-403e-8645-249f3f35ccab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:38:49.423122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a525c-dd98-403e-8645-249f3f35ccab None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:38:49.428584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d194295-e2e2-492f-b1f8-9114c87bc0d7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:38:49.429190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d194295-e2e2-492f-b1f8-9114c87bc0d7 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:49.429467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d194295-e2e2-492f-b1f8-9114c87bc0d7 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:49.429736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d194295-e2e2-492f-b1f8-9114c87bc0d7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:49.438286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d194295-e2e2-492f-b1f8-9114c87bc0d7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:38:49.440475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d194295-e2e2-492f-b1f8-9114c87bc0d7 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:38:49.449987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 757/1513] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:38:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:38:50.304134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d1047-74b3-4ea1-8576-7b86dabacb2f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:50.304699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d1047-74b3-4ea1-8576-7b86dabacb2f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:50.304699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d1047-74b3-4ea1-8576-7b86dabacb2f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:50.305997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 757/1514] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:38:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:38:50.311251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966cfbf1-b977-4ed9-9691-d170f81315b0 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:50.311713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966cfbf1-b977-4ed9-9691-d170f81315b0 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:50.311822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966cfbf1-b977-4ed9-9691-d170f81315b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:50.332246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-966cfbf1-b977-4ed9-9691-d170f81315b0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:38:50.333510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-966cfbf1-b977-4ed9-9691-d170f81315b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:38:50.341610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 758/1515] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:38:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:38:50.496721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4d1047-74b3-4ea1-8576-7b86dabacb2f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:38:50.511528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4d1047-74b3-4ea1-8576-7b86dabacb2f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:38:50.517016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1c1779-ceef-40ac-a311-5dc71f0f9ddc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:38:50.517609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1c1779-ceef-40ac-a311-5dc71f0f9ddc service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:50.517891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1c1779-ceef-40ac-a311-5dc71f0f9ddc service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:50.518165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1c1779-ceef-40ac-a311-5dc71f0f9ddc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:50.526184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1c1779-ceef-40ac-a311-5dc71f0f9ddc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:38:50.527817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1c1779-ceef-40ac-a311-5dc71f0f9ddc service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:38:50.538392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 758/1516] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:38:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:38:51.560499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6de08e-517b-4ce0-825c-9357ac667077 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:51.561100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6de08e-517b-4ce0-825c-9357ac667077 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:51.561100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6de08e-517b-4ce0-825c-9357ac667077 None None] PATH_INFO: `/v3` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:51.562323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 759/1517] 10.222.0.84 () {54 vars in 925 bytes} [Mon Aug 29 22:38:51 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:38:51.568775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055008b7-f13b-44ac-a6b7-df3005ce05db None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:51.569207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055008b7-f13b-44ac-a6b7-df3005ce05db None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:51.569315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055008b7-f13b-44ac-a6b7-df3005ce05db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:51.636905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-055008b7-f13b-44ac-a6b7-df3005ce05db None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:38:51.638120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-055008b7-f13b-44ac-a6b7-df3005ce05db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:38:51.667388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:38:51.727814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 759/1518] 10.222.0.84 () {58 vars in 1015 bytes} [Mon Aug 29 22:38:51 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:38:55.599888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd09a18e-bb4b-45bb-95f8-e47dbc0af01d None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:55.600474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd09a18e-bb4b-45bb-95f8-e47dbc0af01d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:55.600474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd09a18e-bb4b-45bb-95f8-e47dbc0af01d None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:55.601731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 760/1519] 10.222.0.77 () {54 vars in 961 bytes} [Mon Aug 29 22:38:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:38:55.607006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bc223c-f7bc-4aea-bd5d-c2e948ef2173 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:55.607470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bc223c-f7bc-4aea-bd5d-c2e948ef2173 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:55.607576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bc223c-f7bc-4aea-bd5d-c2e948ef2173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:55.626856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b7bc223c-f7bc-4aea-bd5d-c2e948ef2173 None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:38:55.628059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b7bc223c-f7bc-4aea-bd5d-c2e948ef2173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:38:55.636579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 760/1520] 10.222.0.77 () {58 vars in 1058 bytes} [Mon Aug 29 22:38:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:38:55.644508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd09a18e-bb4b-45bb-95f8-e47dbc0af01d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:38:55.658344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd09a18e-bb4b-45bb-95f8-e47dbc0af01d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:38:55.663402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070f3fc4-deff-4ddd-bfaa-c139c2b586bb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:38:55.664021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f3fc4-deff-4ddd-bfaa-c139c2b586bb service placement] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:38:55.664306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f3fc4-deff-4ddd-bfaa-c139c2b586bb service placement] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:38:55.664582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f3fc4-deff-4ddd-bfaa-c139c2b586bb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:38:55.673005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070f3fc4-deff-4ddd-bfaa-c139c2b586bb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:38:55.674598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070f3fc4-deff-4ddd-bfaa-c139c2b586bb service placement] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:38:55.684342 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 761/1521] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:38:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:39:01.775056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be3ecb1-6663-48ac-bffa-b9f3c6f648b4 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:01.775635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be3ecb1-6663-48ac-bffa-b9f3c6f648b4 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:01.775635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be3ecb1-6663-48ac-bffa-b9f3c6f648b4 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:01.776922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 761/1522] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:39:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:39:01.782296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55b08ae-205e-44f1-8323-ae40c38e9efa None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:01.782726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55b08ae-205e-44f1-8323-ae40c38e9efa None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:01.782872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55b08ae-205e-44f1-8323-ae40c38e9efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:01.802100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a55b08ae-205e-44f1-8323-ae40c38e9efa None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:39:01.803321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a55b08ae-205e-44f1-8323-ae40c38e9efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:01.811261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 762/1523] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:39:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:39:01.967096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be3ecb1-6663-48ac-bffa-b9f3c6f648b4 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:39:01.981766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be3ecb1-6663-48ac-bffa-b9f3c6f648b4 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:39:01.987130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3055a7bb-309c-4940-b22a-645d86de9d51 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:39:01.987718 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3055a7bb-309c-4940-b22a-645d86de9d51 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:01.987993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3055a7bb-309c-4940-b22a-645d86de9d51 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:01.988258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3055a7bb-309c-4940-b22a-645d86de9d51 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:01.996324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3055a7bb-309c-4940-b22a-645d86de9d51 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:39:01.997893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3055a7bb-309c-4940-b22a-645d86de9d51 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:39:02.007589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 762/1524] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:39:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:39:02.820776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1830e2e-62c9-4013-a2e3-638a002efbbb None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:02.821271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1830e2e-62c9-4013-a2e3-638a002efbbb None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:02.821367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1830e2e-62c9-4013-a2e3-638a002efbbb None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:02.822676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 763/1525] 127.0.0.1 () {54 vars in 901 bytes} [Mon Aug 29 22:39:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:39:02.827591 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4347dbc-a278-4177-bb80-a82b04d50abd None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:02.828021 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4347dbc-a278-4177-bb80-a82b04d50abd None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:02.828146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4347dbc-a278-4177-bb80-a82b04d50abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:02.848045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e4347dbc-a278-4177-bb80-a82b04d50abd None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:39:02.849234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e4347dbc-a278-4177-bb80-a82b04d50abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:02.857016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 763/1526] 127.0.0.1 () {58 vars in 998 bytes} [Mon Aug 29 22:39:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:39:03.009000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1830e2e-62c9-4013-a2e3-638a002efbbb None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:39:03.023896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1830e2e-62c9-4013-a2e3-638a002efbbb None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:39:03.029650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65beb178-4c59-4331-bf71-7149a2e2c9cd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:39:03.030243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65beb178-4c59-4331-bf71-7149a2e2c9cd service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:03.030521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65beb178-4c59-4331-bf71-7149a2e2c9cd service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:03.030831 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65beb178-4c59-4331-bf71-7149a2e2c9cd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:03.038855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65beb178-4c59-4331-bf71-7149a2e2c9cd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:39:03.040435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65beb178-4c59-4331-bf71-7149a2e2c9cd service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:39:03.050923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 764/1527] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:39:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:39:07.357971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b424aa-3c80-45d4-b432-52354f7f466f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:07.358553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b424aa-3c80-45d4-b432-52354f7f466f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:07.358553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b424aa-3c80-45d4-b432-52354f7f466f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:07.359825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 764/1528] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:39:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:39:07.365614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dd225b-de5a-4719-8eed-0de5fe776da6 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:07.366081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dd225b-de5a-4719-8eed-0de5fe776da6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:07.366178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dd225b-de5a-4719-8eed-0de5fe776da6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:07.386648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c5dd225b-de5a-4719-8eed-0de5fe776da6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:39:07.387887 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c5dd225b-de5a-4719-8eed-0de5fe776da6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:07.396015 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 765/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:39:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:39:07.687260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b424aa-3c80-45d4-b432-52354f7f466f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:39:07.689031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-03b424aa-3c80-45d4-b432-52354f7f466f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:07.768899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b424aa-3c80-45d4-b432-52354f7f466f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:39:07.772514 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df53e32-ea02-4df5-8b7e-28f4965dd0d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:39:07.773110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df53e32-ea02-4df5-8b7e-28f4965dd0d2 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:07.773385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df53e32-ea02-4df5-8b7e-28f4965dd0d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:07.773652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df53e32-ea02-4df5-8b7e-28f4965dd0d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:07.783953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df53e32-ea02-4df5-8b7e-28f4965dd0d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:39:07.785403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df53e32-ea02-4df5-8b7e-28f4965dd0d2 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:39:07.795366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 765/1530] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:39:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:39:09.706595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f64d6-1863-442d-a0e7-ac99a38a4016 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:09.707213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f64d6-1863-442d-a0e7-ac99a38a4016 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:09.707213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302f64d6-1863-442d-a0e7-ac99a38a4016 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:09.708699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 766/1531] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:39:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:39:09.714184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ebcf87-13d8-47a1-be08-92a82f5ca471 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:09.714693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ebcf87-13d8-47a1-be08-92a82f5ca471 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:09.714840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ebcf87-13d8-47a1-be08-92a82f5ca471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:09.735296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-11ebcf87-13d8-47a1-be08-92a82f5ca471 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:39:09.736553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-11ebcf87-13d8-47a1-be08-92a82f5ca471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:09.746464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 766/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:39:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:39:10.035165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302f64d6-1863-442d-a0e7-ac99a38a4016 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:39:10.045810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302f64d6-1863-442d-a0e7-ac99a38a4016 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:39:10.049795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccab6cf5-e42f-4523-8a88-0256cc436792 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:39:10.050376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab6cf5-e42f-4523-8a88-0256cc436792 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:10.050652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab6cf5-e42f-4523-8a88-0256cc436792 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:10.051019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab6cf5-e42f-4523-8a88-0256cc436792 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:10.059775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccab6cf5-e42f-4523-8a88-0256cc436792 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:39:10.061193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccab6cf5-e42f-4523-8a88-0256cc436792 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:39:10.071080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 767/1533] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:39:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:39:13.479867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fdce79-d2f9-4a16-b8de-3a96832e2c04 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:13.480442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fdce79-d2f9-4a16-b8de-3a96832e2c04 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:13.480442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fdce79-d2f9-4a16-b8de-3a96832e2c04 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:13.481737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 767/1534] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:39:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:39:13.487412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f36ca7e-cbd2-4ac0-b2a6-1f541e72f66e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:13.487850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f36ca7e-cbd2-4ac0-b2a6-1f541e72f66e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:13.488017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f36ca7e-cbd2-4ac0-b2a6-1f541e72f66e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:13.507968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7f36ca7e-cbd2-4ac0-b2a6-1f541e72f66e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:39:13.509199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7f36ca7e-cbd2-4ac0-b2a6-1f541e72f66e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:13.516989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 768/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:39:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:39:14.069063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fdce79-d2f9-4a16-b8de-3a96832e2c04 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:39:14.070762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a8fdce79-d2f9-4a16-b8de-3a96832e2c04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:14.135897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:39:14.148969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fdce79-d2f9-4a16-b8de-3a96832e2c04 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:39:14.153741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cc2c6b-f790-4701-975b-1dbf9762c7d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:39:14.154383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cc2c6b-f790-4701-975b-1dbf9762c7d0 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:14.154658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cc2c6b-f790-4701-975b-1dbf9762c7d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:14.154952 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cc2c6b-f790-4701-975b-1dbf9762c7d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:14.164475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cc2c6b-f790-4701-975b-1dbf9762c7d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:39:14.165887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cc2c6b-f790-4701-975b-1dbf9762c7d0 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:39:14.175654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 768/1536] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:39:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:39:16.121887 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573ed4a-b9c8-496c-9492-2a8c3e2c3aed None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:16.122445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573ed4a-b9c8-496c-9492-2a8c3e2c3aed None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:16.122445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573ed4a-b9c8-496c-9492-2a8c3e2c3aed None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:16.123738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 769/1537] 127.0.0.1 () {54 vars in 917 bytes} [Mon Aug 29 22:39:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:39:16.129359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf549848-c276-4c1c-91de-8d5b0624086d None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:16.129799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf549848-c276-4c1c-91de-8d5b0624086d None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:16.129909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf549848-c276-4c1c-91de-8d5b0624086d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:16.149750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-bf549848-c276-4c1c-91de-8d5b0624086d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:39:16.151022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-bf549848-c276-4c1c-91de-8d5b0624086d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:16.159613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 769/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Aug 29 22:39:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:39:16.720333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c573ed4a-b9c8-496c-9492-2a8c3e2c3aed None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:39:16.723322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c573ed4a-b9c8-496c-9492-2a8c3e2c3aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:39:16.736290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c573ed4a-b9c8-496c-9492-2a8c3e2c3aed None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:39:16.739843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9eea2c-4a0b-4211-8214-e0b5880a7323 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:39:16.740437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9eea2c-4a0b-4211-8214-e0b5880a7323 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:39:16.740719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9eea2c-4a0b-4211-8214-e0b5880a7323 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:39:16.740990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9eea2c-4a0b-4211-8214-e0b5880a7323 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:39:16.749868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9eea2c-4a0b-4211-8214-e0b5880a7323 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:39:16.751338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9eea2c-4a0b-4211-8214-e0b5880a7323 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:39:16.760779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 770/1539] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:39:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:39:26.144814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=11 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:39:26.756538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:41.322120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a25c8a-3df6-4399-a4da-18fec035ed1e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.322848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a25c8a-3df6-4399-a4da-18fec035ed1e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.322848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a25c8a-3df6-4399-a4da-18fec035ed1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.351855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a5a25c8a-3df6-4399-a4da-18fec035ed1e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.353489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a5a25c8a-3df6-4399-a4da-18fec035ed1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.374493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 770/1540] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.385501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1068b5-be5a-49be-b451-e397931a2e0f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.386132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1068b5-be5a-49be-b451-e397931a2e0f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.386132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1068b5-be5a-49be-b451-e397931a2e0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.413279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2c1068b5-be5a-49be-b451-e397931a2e0f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.415080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2c1068b5-be5a-49be-b451-e397931a2e0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.429624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 771/1541] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.434955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a25c8a-3df6-4399-a4da-18fec035ed1e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:41.446661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a25c8a-3df6-4399-a4da-18fec035ed1e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:41.459342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:41.459973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.460260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.460536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.461626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:41.461939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:41.463599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:41.491559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 771/1542] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:41.501821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1068b5-be5a-49be-b451-e397931a2e0f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:41.519002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1068b5-be5a-49be-b451-e397931a2e0f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:41.527277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513be55-8200-4998-b56c-d3394dfd9940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:41.528409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513be55-8200-4998-b56c-d3394dfd9940 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.528930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513be55-8200-4998-b56c-d3394dfd9940 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.529434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513be55-8200-4998-b56c-d3394dfd9940 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.537252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513be55-8200-4998-b56c-d3394dfd9940 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:41.539852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513be55-8200-4998-b56c-d3394dfd9940 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:41.605175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 772/1543] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/projects => generated 383 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:41.610228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:41.623024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1224ba65-a811-4393-9707-0a335463976a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.623670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1224ba65-a811-4393-9707-0a335463976a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.624172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1224ba65-a811-4393-9707-0a335463976a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.628828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17553201-2ed3-430a-822b-d2a5a95c2077 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:41.655705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1224ba65-a811-4393-9707-0a335463976a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.657889 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1224ba65-a811-4393-9707-0a335463976a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.748595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5400f37b-2f2a-4282-9429-270889ae707d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:41.752114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5400f37b-2f2a-4282-9429-270889ae707d admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.752683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5400f37b-2f2a-4282-9429-270889ae707d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.753178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5400f37b-2f2a-4282-9429-270889ae707d admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.754034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5400f37b-2f2a-4282-9429-270889ae707d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:41.757687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5400f37b-2f2a-4282-9429-270889ae707d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:41.792557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 772/1544] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/users => generated 292 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:41.794616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b794549b-2433-46d2-bd6a-64df139ec2c8 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.795142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b794549b-2433-46d2-bd6a-64df139ec2c8 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.798836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 773/1545] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:40:41.798836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b88a9-ea5d-472f-857f-a1098bc46b7e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.798836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b88a9-ea5d-472f-857f-a1098bc46b7e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.798836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b88a9-ea5d-472f-857f-a1098bc46b7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.799545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b794549b-2433-46d2-bd6a-64df139ec2c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.824030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b794549b-2433-46d2-bd6a-64df139ec2c8 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.824445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-293b88a9-ea5d-472f-857f-a1098bc46b7e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.825714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b794549b-2433-46d2-bd6a-64df139ec2c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.825856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-293b88a9-ea5d-472f-857f-a1098bc46b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.835209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 773/1546] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.835805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 774/1547] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.836669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f231ee04-5ed4-4d26-9592-373226a25932 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.836957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f231ee04-5ed4-4d26-9592-373226a25932 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.837188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9e32a-8e81-4e63-9e01-dc6f48e1326a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.837298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f231ee04-5ed4-4d26-9592-373226a25932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.837465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9e32a-8e81-4e63-9e01-dc6f48e1326a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.837740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9e32a-8e81-4e63-9e01-dc6f48e1326a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.858639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f231ee04-5ed4-4d26-9592-373226a25932 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.859289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-52d9e32a-8e81-4e63-9e01-dc6f48e1326a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.859908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f231ee04-5ed4-4d26-9592-373226a25932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.860444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-52d9e32a-8e81-4e63-9e01-dc6f48e1326a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.868211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 774/1548] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.868698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 775/1549] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.869016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f231ee04-5ed4-4d26-9592-373226a25932 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:41.869661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd0b5f3-23bf-4769-8c18-0cbae2f3d2d8 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.869943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd0b5f3-23bf-4769-8c18-0cbae2f3d2d8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.870212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd0b5f3-23bf-4769-8c18-0cbae2f3d2d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.882679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f231ee04-5ed4-4d26-9592-373226a25932 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:41.888188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f29f2b-0241-474a-a99b-dfc999cb6848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:41.888805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f29f2b-0241-474a-a99b-dfc999cb6848 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.889120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f29f2b-0241-474a-a99b-dfc999cb6848 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.889362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f29f2b-0241-474a-a99b-dfc999cb6848 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.890012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f29f2b-0241-474a-a99b-dfc999cb6848 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:41.892185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f29f2b-0241-474a-a99b-dfc999cb6848 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:41.896501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1dd0b5f3-23bf-4769-8c18-0cbae2f3d2d8 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.897699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1dd0b5f3-23bf-4769-8c18-0cbae2f3d2d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.902707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 775/1550] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:41 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:41.904017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f09d55-ff06-4cf6-9c0d-d1a18face5b1 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.904305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f09d55-ff06-4cf6-9c0d-d1a18face5b1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.904622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f09d55-ff06-4cf6-9c0d-d1a18face5b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.917006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 776/1551] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.919641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf8820-a024-4c66-b5f9-817de76a9197 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.919920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf8820-a024-4c66-b5f9-817de76a9197 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.920231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf8820-a024-4c66-b5f9-817de76a9197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.929929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d2f09d55-ff06-4cf6-9c0d-d1a18face5b1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.931397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d2f09d55-ff06-4cf6-9c0d-d1a18face5b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.943196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-03cf8820-a024-4c66-b5f9-817de76a9197 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.943380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 776/1552] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.944399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-03cf8820-a024-4c66-b5f9-817de76a9197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.944617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79a8cba-0021-4d97-a009-80e871626d6d None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.944909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79a8cba-0021-4d97-a009-80e871626d6d None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.945188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79a8cba-0021-4d97-a009-80e871626d6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.959504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 777/1553] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.960250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415c011-3149-4e84-827f-9ff660b78d09 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.960532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415c011-3149-4e84-827f-9ff660b78d09 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.960802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415c011-3149-4e84-827f-9ff660b78d09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.975492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a79a8cba-0021-4d97-a009-80e871626d6d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.976723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a79a8cba-0021-4d97-a009-80e871626d6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.983400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d415c011-3149-4e84-827f-9ff660b78d09 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:41.984587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d415c011-3149-4e84-827f-9ff660b78d09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:41.985406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 777/1554] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.986665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627adba7-34b3-4e62-a34a-c946379a0bd2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.986988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627adba7-34b3-4e62-a34a-c946379a0bd2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.987265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627adba7-34b3-4e62-a34a-c946379a0bd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:41.994069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 778/1555] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:41.998867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647e3cd6-3bba-4557-bb54-68f2e86c6a1b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:41.999105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647e3cd6-3bba-4557-bb54-68f2e86c6a1b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:41.999386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647e3cd6-3bba-4557-bb54-68f2e86c6a1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.007400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-627adba7-34b3-4e62-a34a-c946379a0bd2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:42.008601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-627adba7-34b3-4e62-a34a-c946379a0bd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:42.017316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 778/1556] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:42.017931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627adba7-34b3-4e62-a34a-c946379a0bd2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.021011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-647e3cd6-3bba-4557-bb54-68f2e86c6a1b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:42.022073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-647e3cd6-3bba-4557-bb54-68f2e86c6a1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:42.027584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627adba7-34b3-4e62-a34a-c946379a0bd2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.030074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 779/1557] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:42.030899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647e3cd6-3bba-4557-bb54-68f2e86c6a1b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.035663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdf0962-9eed-405f-96d4-95360256ed0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.036637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf0962-9eed-405f-96d4-95360256ed0d admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.036637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf0962-9eed-405f-96d4-95360256ed0d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.036852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf0962-9eed-405f-96d4-95360256ed0d admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.037431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdf0962-9eed-405f-96d4-95360256ed0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.039058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdf0962-9eed-405f-96d4-95360256ed0d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.041018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647e3cd6-3bba-4557-bb54-68f2e86c6a1b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.045714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71123415-ac74-4222-b9f4-059c8115236a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.046318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71123415-ac74-4222-b9f4-059c8115236a admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.046657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71123415-ac74-4222-b9f4-059c8115236a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.046974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71123415-ac74-4222-b9f4-059c8115236a admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.047620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71123415-ac74-4222-b9f4-059c8115236a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:42.047935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71123415-ac74-4222-b9f4-059c8115236a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.048982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING py.warnings [None req-71123415-ac74-4222-b9f4-059c8115236a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:40:42.048982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: warnings.warn(msg) Aug 29 22:40:42.048982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: Aug 29 22:40:42.049257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 779/1558] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:42.049869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71123415-ac74-4222-b9f4-059c8115236a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.052944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40331d-db44-47a2-9fd1-65814c760320 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.053239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40331d-db44-47a2-9fd1-65814c760320 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.053516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40331d-db44-47a2-9fd1-65814c760320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.066423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 780/1559] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:42.066423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b260533-ad2f-401b-8988-45020742e103 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.066423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b260533-ad2f-401b-8988-45020742e103 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.066423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b260533-ad2f-401b-8988-45020742e103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.079080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1f40331d-db44-47a2-9fd1-65814c760320 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:42.080303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1f40331d-db44-47a2-9fd1-65814c760320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:42.085468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1b260533-ad2f-401b-8988-45020742e103 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:42.086671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1b260533-ad2f-401b-8988-45020742e103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:42.088724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 780/1560] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:42.089541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f40331d-db44-47a2-9fd1-65814c760320 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.096367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 781/1561] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:42.097193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b260533-ad2f-401b-8988-45020742e103 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.100366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f40331d-db44-47a2-9fd1-65814c760320 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.105044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.105647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.105925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.106197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.106882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:42.107192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.108800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.109896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b260533-ad2f-401b-8988-45020742e103 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.114655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.115283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.115569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.115845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.116496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:42.116807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.118397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.121086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 781/1562] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:42.128637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.134844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 782/1563] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:42.134844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.142175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4824c3-469b-451b-b18f-da5e8f58f988 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.143742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80413712-faef-49da-bee9-68ac72f9b52b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.147136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.147742 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.148020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.148290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.148462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.148943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:42.149071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.149242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.149354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.149570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.150216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:42.150525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.150899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.152157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.164596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 783/1565] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:42.164823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 782/1565] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:42.165420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.166083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.177069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df24d457-7416-47ce-875e-839d33ac6385 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.181751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.181933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc2606e-3e87-4f8a-94cf-6a3e8a228c07 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.182353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.182633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.183569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.184211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.185255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING py.warnings [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:40:42.185255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: warnings.warn(msg) Aug 29 22:40:42.185255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: Aug 29 22:40:42.186400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da9753-3fb1-4220-939d-ef162a502134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.186533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.187090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da9753-3fb1-4220-939d-ef162a502134 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.187920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da9753-3fb1-4220-939d-ef162a502134 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.188197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da9753-3fb1-4220-939d-ef162a502134 admin admin] PATH_INFO: `/v3/projects/ff9bc95c0e074f66a511f524154b70bf/users/298bfaf3cc0d478093d3ee97a3c1e734/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.205774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81da9753-3fb1-4220-939d-ef162a502134 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff9bc95c0e074f66a511f524154b70bf, user_id=298bfaf3cc0d478093d3ee97a3c1e734, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.208537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81da9753-3fb1-4220-939d-ef162a502134 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.214972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 783/1566] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/projects => generated 403 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.214972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.226004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 784/1567] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:42 2022] PUT /identity/v3/projects/ff9bc95c0e074f66a511f524154b70bf/users/298bfaf3cc0d478093d3ee97a3c1e734/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:42.226946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da9753-3fb1-4220-939d-ef162a502134 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.240382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:42.242631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:42.287791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a0e054-7879-4574-84fb-0c171fa54b9f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.291907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81da9753-3fb1-4220-939d-ef162a502134 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.292279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.292883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.293156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.293426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.294064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:42.294364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.295995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.297221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.298036 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.298261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.298534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.299226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:42.299531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.301138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.309684 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 784/1568] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:42.310446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.313152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 785/1569] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:42.313943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.323744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69bf8ad-8ce5-496d-a92c-8d00da5e50d3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.328964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028aa67c-237a-4ee5-878e-5efc99d5537b admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.329653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff515b-00f8-4b59-a944-d320ff91b883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.329653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff515b-00f8-4b59-a944-d320ff91b883 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.329865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff515b-00f8-4b59-a944-d320ff91b883 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.330017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff515b-00f8-4b59-a944-d320ff91b883 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.330660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ff515b-00f8-4b59-a944-d320ff91b883 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.332606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ff515b-00f8-4b59-a944-d320ff91b883 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.333611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bac97-4d21-4768-865f-b7073c2e9a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.334209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bac97-4d21-4768-865f-b7073c2e9a6f admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.334490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bac97-4d21-4768-865f-b7073c2e9a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.334794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bac97-4d21-4768-865f-b7073c2e9a6f admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.335433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690bac97-4d21-4768-865f-b7073c2e9a6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.337341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690bac97-4d21-4768-865f-b7073c2e9a6f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.362512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 786/1570] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.363553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff515b-00f8-4b59-a944-d320ff91b883 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.369369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 785/1571] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/projects => generated 381 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.370076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bac97-4d21-4768-865f-b7073c2e9a6f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.428728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff515b-00f8-4b59-a944-d320ff91b883 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.440834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bac97-4d21-4768-865f-b7073c2e9a6f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.447364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fad8080-e47f-4d5c-b119-4d42fe756db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.448007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fad8080-e47f-4d5c-b119-4d42fe756db8 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.448434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fad8080-e47f-4d5c-b119-4d42fe756db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.448711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fad8080-e47f-4d5c-b119-4d42fe756db8 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.449389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fad8080-e47f-4d5c-b119-4d42fe756db8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.451401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fad8080-e47f-4d5c-b119-4d42fe756db8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.481032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 786/1572] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/projects => generated 365 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.481849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fad8080-e47f-4d5c-b119-4d42fe756db8 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.491207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8542318-f07f-4e03-b472-55eaaf9e1149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.491819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8542318-f07f-4e03-b472-55eaaf9e1149 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.492095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8542318-f07f-4e03-b472-55eaaf9e1149 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.492370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8542318-f07f-4e03-b472-55eaaf9e1149 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.493001 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8542318-f07f-4e03-b472-55eaaf9e1149 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.494963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8542318-f07f-4e03-b472-55eaaf9e1149 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.524612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 787/1573] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/projects => generated 389 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.525965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b3ce4-8c31-4d2f-a86e-a9f85643c9a4 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.526251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b3ce4-8c31-4d2f-a86e-a9f85643c9a4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.526529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b3ce4-8c31-4d2f-a86e-a9f85643c9a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.549693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fad8080-e47f-4d5c-b119-4d42fe756db8 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.554799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-173b3ce4-8c31-4d2f-a86e-a9f85643c9a4 None None] MFA Rules not processed for user `298bfaf3cc0d478093d3ee97a3c1e734`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:42.556008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-173b3ce4-8c31-4d2f-a86e-a9f85643c9a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:42.561114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:42.571702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:42.611521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e68323-eba1-4f6e-a864-f2db29555181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.612125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e68323-eba1-4f6e-a864-f2db29555181 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.612400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e68323-eba1-4f6e-a864-f2db29555181 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.612668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e68323-eba1-4f6e-a864-f2db29555181 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.613295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e68323-eba1-4f6e-a864-f2db29555181 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.615290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e68323-eba1-4f6e-a864-f2db29555181 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.623874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 788/1574] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:42.624969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b3ce4-8c31-4d2f-a86e-a9f85643c9a4 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.642770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 787/1575] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/users => generated 302 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.643514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b3ce4-8c31-4d2f-a86e-a9f85643c9a4 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.643740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e68323-eba1-4f6e-a864-f2db29555181 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.648288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ead0fc9-b81d-4fbb-a899-baa0de85f830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.649034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ead0fc9-b81d-4fbb-a899-baa0de85f830 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.649430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ead0fc9-b81d-4fbb-a899-baa0de85f830 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.649768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ead0fc9-b81d-4fbb-a899-baa0de85f830 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.650472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ead0fc9-b81d-4fbb-a899-baa0de85f830 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.652292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e68323-eba1-4f6e-a864-f2db29555181 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.652685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ead0fc9-b81d-4fbb-a899-baa0de85f830 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.658017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e67f6d2-50a0-4962-8586-fa82e79ce1bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.659031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e67f6d2-50a0-4962-8586-fa82e79ce1bc admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.659262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e67f6d2-50a0-4962-8586-fa82e79ce1bc admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.659614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e67f6d2-50a0-4962-8586-fa82e79ce1bc admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.660439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e67f6d2-50a0-4962-8586-fa82e79ce1bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.662473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e67f6d2-50a0-4962-8586-fa82e79ce1bc admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.693596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 789/1576] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/projects => generated 379 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.693596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ead0fc9-b81d-4fbb-a899-baa0de85f830 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.696358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 788/1577] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/projects => generated 373 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.697620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e67f6d2-50a0-4962-8586-fa82e79ce1bc admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.757374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ead0fc9-b81d-4fbb-a899-baa0de85f830 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.759187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e67f6d2-50a0-4962-8586-fa82e79ce1bc admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.762120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.762787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.763111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.763405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.763636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e22843-3848-4635-b1e9-6e806a34c4ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.764149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.764345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e22843-3848-4635-b1e9-6e806a34c4ee admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.764620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e22843-3848-4635-b1e9-6e806a34c4ee admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.764890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e22843-3848-4635-b1e9-6e806a34c4ee admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.765063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING py.warnings [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:40:42.765063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: warnings.warn(msg) Aug 29 22:40:42.765063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: Aug 29 22:40:42.765549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e22843-3848-4635-b1e9-6e806a34c4ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.766283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.767490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e22843-3848-4635-b1e9-6e806a34c4ee admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.789668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 789/1578] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/users => generated 291 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.790485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e22843-3848-4635-b1e9-6e806a34c4ee admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.790704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 790/1579] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/users => generated 288 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.791543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.805540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814b1940-0e4c-4685-800b-bba8ed8ae075 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.806923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e22843-3848-4635-b1e9-6e806a34c4ee admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.810127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5252d1e5-1504-4ae3-968d-2629b0c98afa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.810766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5252d1e5-1504-4ae3-968d-2629b0c98afa admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.811045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5252d1e5-1504-4ae3-968d-2629b0c98afa admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.811322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5252d1e5-1504-4ae3-968d-2629b0c98afa admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.811562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1cdac1-a0b1-4365-9d2a-ce9ae9447ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.811960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5252d1e5-1504-4ae3-968d-2629b0c98afa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.812185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1cdac1-a0b1-4365-9d2a-ce9ae9447ec7 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.813054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1cdac1-a0b1-4365-9d2a-ce9ae9447ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.813341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1cdac1-a0b1-4365-9d2a-ce9ae9447ec7 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.813913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5252d1e5-1504-4ae3-968d-2629b0c98afa admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.814018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1cdac1-a0b1-4365-9d2a-ce9ae9447ec7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.819002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1cdac1-a0b1-4365-9d2a-ce9ae9447ec7 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.840844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 790/1580] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/users => generated 283 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.840844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1cdac1-a0b1-4365-9d2a-ce9ae9447ec7 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.841426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 791/1581] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/users => generated 295 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.842647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa7948-b31c-47f1-a735-54edcef32932 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.842965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa7948-b31c-47f1-a735-54edcef32932 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.843252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa7948-b31c-47f1-a735-54edcef32932 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.845022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 792/1582] 10.222.0.77 () {54 vars in 962 bytes} [Mon Aug 29 22:40:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:40:42.845983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aa7948-b31c-47f1-a735-54edcef32932 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.853280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1cdac1-a0b1-4365-9d2a-ce9ae9447ec7 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.857954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faeb9019-26a7-4872-bcff-755512c78751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.858555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faeb9019-26a7-4872-bcff-755512c78751 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.858864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faeb9019-26a7-4872-bcff-755512c78751 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.859944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aa7948-b31c-47f1-a735-54edcef32932 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.860065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faeb9019-26a7-4872-bcff-755512c78751 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.860562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faeb9019-26a7-4872-bcff-755512c78751 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.862159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faeb9019-26a7-4872-bcff-755512c78751 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.865887 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b5b382-f0c1-4173-9d5f-910d7ef2f1ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.866492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b5b382-f0c1-4173-9d5f-910d7ef2f1ba admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.867006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b5b382-f0c1-4173-9d5f-910d7ef2f1ba admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.867283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b5b382-f0c1-4173-9d5f-910d7ef2f1ba admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.867946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b5b382-f0c1-4173-9d5f-910d7ef2f1ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.869894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b5b382-f0c1-4173-9d5f-910d7ef2f1ba admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.873623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 791/1583] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:42.874531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faeb9019-26a7-4872-bcff-755512c78751 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.885750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faeb9019-26a7-4872-bcff-755512c78751 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.890703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27175f5a-d6e9-4acb-95ef-dade082254d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.891447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27175f5a-d6e9-4acb-95ef-dade082254d4 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.891728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27175f5a-d6e9-4acb-95ef-dade082254d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.891997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27175f5a-d6e9-4acb-95ef-dade082254d4 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.892720 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27175f5a-d6e9-4acb-95ef-dade082254d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.894663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27175f5a-d6e9-4acb-95ef-dade082254d4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.894852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 793/1584] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/users => generated 290 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.895902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b5b382-f0c1-4173-9d5f-910d7ef2f1ba admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.906023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b5b382-f0c1-4173-9d5f-910d7ef2f1ba admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.912435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f668e97-8181-4d49-80b9-277d0080a695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.913376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f668e97-8181-4d49-80b9-277d0080a695 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.913970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f668e97-8181-4d49-80b9-277d0080a695 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.914250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f668e97-8181-4d49-80b9-277d0080a695 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.915008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f668e97-8181-4d49-80b9-277d0080a695 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.916619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f668e97-8181-4d49-80b9-277d0080a695 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.921379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 792/1585] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/users => generated 287 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:42.921379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27175f5a-d6e9-4acb-95ef-dade082254d4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.929024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 794/1586] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:42.929877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f668e97-8181-4d49-80b9-277d0080a695 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.929981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27175f5a-d6e9-4acb-95ef-dade082254d4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.935168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c639487-29e3-48f4-80d9-67346c9d0bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.935765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c639487-29e3-48f4-80d9-67346c9d0bc3 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.936872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c639487-29e3-48f4-80d9-67346c9d0bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.936872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c639487-29e3-48f4-80d9-67346c9d0bc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.937235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c639487-29e3-48f4-80d9-67346c9d0bc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.938619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c639487-29e3-48f4-80d9-67346c9d0bc3 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.940107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f668e97-8181-4d49-80b9-277d0080a695 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.946765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b4806d-db46-473b-bfb0-90fde7c717f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.947619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4806d-db46-473b-bfb0-90fde7c717f5 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.948017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4806d-db46-473b-bfb0-90fde7c717f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.948431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4806d-db46-473b-bfb0-90fde7c717f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.949219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b4806d-db46-473b-bfb0-90fde7c717f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.949351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 793/1587] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:42.950090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c639487-29e3-48f4-80d9-67346c9d0bc3 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:42.950885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b4806d-db46-473b-bfb0-90fde7c717f5 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.962707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 795/1588] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:42.963275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68306605-9163-4d48-ab61-add69cfebd2d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.963412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c639487-29e3-48f4-80d9-67346c9d0bc3 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:42.963601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68306605-9163-4d48-ab61-add69cfebd2d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.964068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68306605-9163-4d48-ab61-add69cfebd2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.969743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019e4235-14ca-49a4-88cb-9b7871c51e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:42.970620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019e4235-14ca-49a4-88cb-9b7871c51e1d admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:42.970937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019e4235-14ca-49a4-88cb-9b7871c51e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:42.971386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019e4235-14ca-49a4-88cb-9b7871c51e1d admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:42.972055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019e4235-14ca-49a4-88cb-9b7871c51e1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:42.973665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019e4235-14ca-49a4-88cb-9b7871c51e1d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:42.986754 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 794/1589] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:42.990635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019e4235-14ca-49a4-88cb-9b7871c51e1d admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.000497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019e4235-14ca-49a4-88cb-9b7871c51e1d admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.003626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:43.005210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb137ec5-7b5f-4058-90bf-4456617c5251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.005820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb137ec5-7b5f-4058-90bf-4456617c5251 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.006105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb137ec5-7b5f-4058-90bf-4456617c5251 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.006380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb137ec5-7b5f-4058-90bf-4456617c5251 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.007071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb137ec5-7b5f-4058-90bf-4456617c5251 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.008683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb137ec5-7b5f-4058-90bf-4456617c5251 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.013652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-68306605-9163-4d48-ab61-add69cfebd2d None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:43.014915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-68306605-9163-4d48-ab61-add69cfebd2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:43.024877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 795/1590] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.024877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb137ec5-7b5f-4058-90bf-4456617c5251 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.026139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=87,keystone:INSERT=13 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:43.036258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb137ec5-7b5f-4058-90bf-4456617c5251 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.041359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f89053-b241-4521-b189-f10c334a695c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.042003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f89053-b241-4521-b189-f10c334a695c admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.042311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f89053-b241-4521-b189-f10c334a695c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.042632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f89053-b241-4521-b189-f10c334a695c admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.049946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f89053-b241-4521-b189-f10c334a695c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.049946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f89053-b241-4521-b189-f10c334a695c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.056253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 796/1591] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.057028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f89053-b241-4521-b189-f10c334a695c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.067159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f89053-b241-4521-b189-f10c334a695c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.072014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2990d918-dbdd-4ec3-a2a2-dabf5292d7f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.072621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2990d918-dbdd-4ec3-a2a2-dabf5292d7f8 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.072898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2990d918-dbdd-4ec3-a2a2-dabf5292d7f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.073167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2990d918-dbdd-4ec3-a2a2-dabf5292d7f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.073813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2990d918-dbdd-4ec3-a2a2-dabf5292d7f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.075445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2990d918-dbdd-4ec3-a2a2-dabf5292d7f8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.088323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 797/1592] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.089539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2990d918-dbdd-4ec3-a2a2-dabf5292d7f8 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.100185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2990d918-dbdd-4ec3-a2a2-dabf5292d7f8 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.103129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 796/1593] 10.222.0.77 () {58 vars in 1059 bytes} [Mon Aug 29 22:40:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:43.104141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68306605-9163-4d48-ab61-add69cfebd2d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.109532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dc1467-6d84-4252-8091-d622509265ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.110803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dc1467-6d84-4252-8091-d622509265ea admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.111093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dc1467-6d84-4252-8091-d622509265ea admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.111367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dc1467-6d84-4252-8091-d622509265ea admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.112022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dc1467-6d84-4252-8091-d622509265ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.113629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dc1467-6d84-4252-8091-d622509265ea admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.114515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68306605-9163-4d48-ab61-add69cfebd2d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.119483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2997a05-7b5c-4928-b3e3-f265bc34f549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.119816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2997a05-7b5c-4928-b3e3-f265bc34f549 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.120098 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2997a05-7b5c-4928-b3e3-f265bc34f549 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.120370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2997a05-7b5c-4928-b3e3-f265bc34f549 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.121048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2997a05-7b5c-4928-b3e3-f265bc34f549 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.122668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2997a05-7b5c-4928-b3e3-f265bc34f549 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.127515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 798/1594] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.127515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dc1467-6d84-4252-8091-d622509265ea admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.135558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dc1467-6d84-4252-8091-d622509265ea admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.137276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 797/1595] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.138446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2997a05-7b5c-4928-b3e3-f265bc34f549 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.142838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d75cd-daa0-40ce-a9e7-9b87ada96198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.143459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d75cd-daa0-40ce-a9e7-9b87ada96198 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.143738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d75cd-daa0-40ce-a9e7-9b87ada96198 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.144013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d75cd-daa0-40ce-a9e7-9b87ada96198 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.144850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d75cd-daa0-40ce-a9e7-9b87ada96198 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.146465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d75cd-daa0-40ce-a9e7-9b87ada96198 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.149959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2997a05-7b5c-4928-b3e3-f265bc34f549 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.155615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad1057e-543b-46e7-8f0a-09789d4f5973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.156855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad1057e-543b-46e7-8f0a-09789d4f5973 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.157126 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad1057e-543b-46e7-8f0a-09789d4f5973 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.157390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad1057e-543b-46e7-8f0a-09789d4f5973 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.158053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad1057e-543b-46e7-8f0a-09789d4f5973 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.158807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 799/1596] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.159721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d75cd-daa0-40ce-a9e7-9b87ada96198 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.162986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad1057e-543b-46e7-8f0a-09789d4f5973 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.169860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d75cd-daa0-40ce-a9e7-9b87ada96198 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.174011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 798/1597] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.175014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad1057e-543b-46e7-8f0a-09789d4f5973 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.175250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.176138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.176433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.176663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] PATH_INFO: `/v3/projects/b847982223024b6dadd05dde4b7a3b29/users/ad16cd3ff11e41428db531180e088338/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.185087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad1057e-543b-46e7-8f0a-09789d4f5973 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.189941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09643a51-75e6-43d4-9f18-82a60b9127ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.190539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09643a51-75e6-43d4-9f18-82a60b9127ce admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.190883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09643a51-75e6-43d4-9f18-82a60b9127ce admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.191112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09643a51-75e6-43d4-9f18-82a60b9127ce admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.191786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09643a51-75e6-43d4-9f18-82a60b9127ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.192853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=b847982223024b6dadd05dde4b7a3b29, user_id=ad16cd3ff11e41428db531180e088338, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.193387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09643a51-75e6-43d4-9f18-82a60b9127ce admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.194075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING py.warnings [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:40:43.194075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: warnings.warn(msg) Aug 29 22:40:43.194075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: Aug 29 22:40:43.196095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.206056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 799/1598] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.206992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09643a51-75e6-43d4-9f18-82a60b9127ce admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.214254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 800/1599] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:43 2022] PUT /identity/v3/projects/b847982223024b6dadd05dde4b7a3b29/users/ad16cd3ff11e41428db531180e088338/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:43.214985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.218436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09643a51-75e6-43d4-9f18-82a60b9127ce admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.288529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca5d10-f778-456f-b4d7-7bbe690aef80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.289201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca5d10-f778-456f-b4d7-7bbe690aef80 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.289478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca5d10-f778-456f-b4d7-7bbe690aef80 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.289748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca5d10-f778-456f-b4d7-7bbe690aef80 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.290395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ca5d10-f778-456f-b4d7-7bbe690aef80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.292084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ca5d10-f778-456f-b4d7-7bbe690aef80 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.299803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1696f4d4-63ee-4639-9727-82935034a9ed admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.306172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 800/1600] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:43.306172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca5d10-f778-456f-b4d7-7bbe690aef80 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.306492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd832cb-4b1d-4656-a3ac-986c7be1ae32 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.306698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd832cb-4b1d-4656-a3ac-986c7be1ae32 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.307054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd832cb-4b1d-4656-a3ac-986c7be1ae32 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.307308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd832cb-4b1d-4656-a3ac-986c7be1ae32 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.314047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca5d10-f778-456f-b4d7-7bbe690aef80 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.318977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f028709c-0e40-4f75-8fd6-64fbc29f00d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.319679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028709c-0e40-4f75-8fd6-64fbc29f00d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.319919 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028709c-0e40-4f75-8fd6-64fbc29f00d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.320188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028709c-0e40-4f75-8fd6-64fbc29f00d9 admin admin] PATH_INFO: `/v3/projects/1fcb391184b94306898e43379094e5f6/users/cc7aa14777ab43a083b3a120c4349d67/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.337121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f028709c-0e40-4f75-8fd6-64fbc29f00d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fcb391184b94306898e43379094e5f6, user_id=cc7aa14777ab43a083b3a120c4349d67, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.339872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f028709c-0e40-4f75-8fd6-64fbc29f00d9 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.362938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 801/1601] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:43 2022] PUT /identity/v3/projects/1fcb391184b94306898e43379094e5f6/users/cc7aa14777ab43a083b3a120c4349d67/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:43.367945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f028709c-0e40-4f75-8fd6-64fbc29f00d9 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.371292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd832cb-4b1d-4656-a3ac-986c7be1ae32 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.372905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd832cb-4b1d-4656-a3ac-986c7be1ae32 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.382310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 801/1602] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:43.383115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd832cb-4b1d-4656-a3ac-986c7be1ae32 service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.396578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd832cb-4b1d-4656-a3ac-986c7be1ae32 service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.419229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:43.430645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:43.448303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f028709c-0e40-4f75-8fd6-64fbc29f00d9 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.454686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b90f08-578b-4263-a806-cc48fa6c0d5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.455448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b90f08-578b-4263-a806-cc48fa6c0d5a admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.455728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b90f08-578b-4263-a806-cc48fa6c0d5a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.456397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b90f08-578b-4263-a806-cc48fa6c0d5a admin admin] PATH_INFO: `/v3/projects/cf0dd257dea04aca845b17f51e1c200e/users/70da750fbe764462b50dc0fcef3eb0e6/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.468224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73daed7a-0b28-4e8f-a173-6bbd35fb8a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.469589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73daed7a-0b28-4e8f-a173-6bbd35fb8a04 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.469881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73daed7a-0b28-4e8f-a173-6bbd35fb8a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.470166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73daed7a-0b28-4e8f-a173-6bbd35fb8a04 admin admin] PATH_INFO: `/v3/projects/65c54ce7f3d24882bfb25652bad3c324/users/12cb4bbbe2d1439cabd3a6c3c3de9484/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.475065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b90f08-578b-4263-a806-cc48fa6c0d5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf0dd257dea04aca845b17f51e1c200e, user_id=70da750fbe764462b50dc0fcef3eb0e6, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.477808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b90f08-578b-4263-a806-cc48fa6c0d5a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.487922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73daed7a-0b28-4e8f-a173-6bbd35fb8a04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65c54ce7f3d24882bfb25652bad3c324, user_id=12cb4bbbe2d1439cabd3a6c3c3de9484, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.490830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73daed7a-0b28-4e8f-a173-6bbd35fb8a04 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.493327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 802/1603] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:43 2022] PUT /identity/v3/projects/cf0dd257dea04aca845b17f51e1c200e/users/70da750fbe764462b50dc0fcef3eb0e6/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:43.493327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b90f08-578b-4263-a806-cc48fa6c0d5a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.514638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 802/1604] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:43 2022] PUT /identity/v3/projects/65c54ce7f3d24882bfb25652bad3c324/users/12cb4bbbe2d1439cabd3a6c3c3de9484/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:43.516391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73daed7a-0b28-4e8f-a173-6bbd35fb8a04 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.572074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b90f08-578b-4263-a806-cc48fa6c0d5a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.606354 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73daed7a-0b28-4e8f-a173-6bbd35fb8a04 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.611907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d1c46f-75a8-49ff-b896-02ede3342455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.613038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1c46f-75a8-49ff-b896-02ede3342455 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.613391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1c46f-75a8-49ff-b896-02ede3342455 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.613753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1c46f-75a8-49ff-b896-02ede3342455 admin admin] PATH_INFO: `/v3/projects/5b5cee4eb87f4428aa0c68502f2eaab1/users/72c45a0b5705455c8bb3f09053e325cf/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.629606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d1c46f-75a8-49ff-b896-02ede3342455 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b5cee4eb87f4428aa0c68502f2eaab1, user_id=72c45a0b5705455c8bb3f09053e325cf, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.633174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d1c46f-75a8-49ff-b896-02ede3342455 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.646418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 803/1605] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:43 2022] PUT /identity/v3/projects/5b5cee4eb87f4428aa0c68502f2eaab1/users/72c45a0b5705455c8bb3f09053e325cf/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:43.647692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1fe33e-d4cc-4cc2-a774-b6fb18f7a3e0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.647977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1fe33e-d4cc-4cc2-a774-b6fb18f7a3e0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.648249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1fe33e-d4cc-4cc2-a774-b6fb18f7a3e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.650985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dc9ea3-7894-4591-810b-9015ddce5c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.651913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc9ea3-7894-4591-810b-9015ddce5c95 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.652201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc9ea3-7894-4591-810b-9015ddce5c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.652523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc9ea3-7894-4591-810b-9015ddce5c95 admin admin] PATH_INFO: `/v3/projects/05e6d1afb63446af887a1de133b7bf2e/users/003dd6612cda48618966db8c0788b794/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.668564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5dc9ea3-7894-4591-810b-9015ddce5c95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05e6d1afb63446af887a1de133b7bf2e, user_id=003dd6612cda48618966db8c0788b794, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.671343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5dc9ea3-7894-4591-810b-9015ddce5c95 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.679005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1c1fe33e-d4cc-4cc2-a774-b6fb18f7a3e0 None None] MFA Rules not processed for user `ad16cd3ff11e41428db531180e088338`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:43.680542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1c1fe33e-d4cc-4cc2-a774-b6fb18f7a3e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:43.685565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 803/1606] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:43 2022] PUT /identity/v3/projects/05e6d1afb63446af887a1de133b7bf2e/users/003dd6612cda48618966db8c0788b794/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:43.686152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dc9ea3-7894-4591-810b-9015ddce5c95 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.735693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:43.750866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 804/1607] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:43 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:43.752373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4cc09c-63da-49e1-b37a-b6ed6c8d83a1 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.752660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4cc09c-63da-49e1-b37a-b6ed6c8d83a1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.752932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4cc09c-63da-49e1-b37a-b6ed6c8d83a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.754852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dc9ea3-7894-4591-810b-9015ddce5c95 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.759698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279152da-e3dd-42ad-9a34-ab1264098f37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.760506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279152da-e3dd-42ad-9a34-ab1264098f37 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.760823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279152da-e3dd-42ad-9a34-ab1264098f37 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.761240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279152da-e3dd-42ad-9a34-ab1264098f37 admin admin] PATH_INFO: `/v3/projects/31484448939c42e39d5977a1c8ccae52/users/b34db43f86264c448bf98693bfed7204/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.771579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-0b4cc09c-63da-49e1-b37a-b6ed6c8d83a1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:43.773433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0b4cc09c-63da-49e1-b37a-b6ed6c8d83a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:43.776663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279152da-e3dd-42ad-9a34-ab1264098f37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=31484448939c42e39d5977a1c8ccae52, user_id=b34db43f86264c448bf98693bfed7204, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.780343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279152da-e3dd-42ad-9a34-ab1264098f37 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.783728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 805/1608] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:43.788217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d8e9c-d971-4f4e-aefb-94293aa63f53 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.789223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d8e9c-d971-4f4e-aefb-94293aa63f53 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.789394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d8e9c-d971-4f4e-aefb-94293aa63f53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.799024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 804/1609] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:43 2022] PUT /identity/v3/projects/31484448939c42e39d5977a1c8ccae52/users/b34db43f86264c448bf98693bfed7204/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:43.800521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab86f4-57bd-48d0-a38d-00c42d768833 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.800995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab86f4-57bd-48d0-a38d-00c42d768833 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.801425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab86f4-57bd-48d0-a38d-00c42d768833 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.811177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f29d8e9c-d971-4f4e-aefb-94293aa63f53 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:43.812498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f29d8e9c-d971-4f4e-aefb-94293aa63f53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:43.830938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:43.832139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-efab86f4-57bd-48d0-a38d-00c42d768833 None None] MFA Rules not processed for user `12cb4bbbe2d1439cabd3a6c3c3de9484`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:43.833550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-efab86f4-57bd-48d0-a38d-00c42d768833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:43.882298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 806/1610] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:43.883683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1324f279-2bea-42a8-8b6b-93b1c89b475f None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.883998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1324f279-2bea-42a8-8b6b-93b1c89b475f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.884274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1324f279-2bea-42a8-8b6b-93b1c89b475f None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.885804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 807/1611] 10.222.0.77 () {54 vars in 912 bytes} [Mon Aug 29 22:40:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:40:43.886702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f021fe08-649f-4a68-84dd-6202c9ba7020 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.887046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f021fe08-649f-4a68-84dd-6202c9ba7020 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.887331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f021fe08-649f-4a68-84dd-6202c9ba7020 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.907263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f021fe08-649f-4a68-84dd-6202c9ba7020 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:43.908474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f021fe08-649f-4a68-84dd-6202c9ba7020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:43.908616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 805/1612] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:43 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:43.913754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b892f-7e7f-4000-9dd3-dd6f84ee1fa6 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.914047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b892f-7e7f-4000-9dd3-dd6f84ee1fa6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.914690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b892f-7e7f-4000-9dd3-dd6f84ee1fa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.917291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 808/1613] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:43.918076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f021fe08-649f-4a68-84dd-6202c9ba7020 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.928969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f021fe08-649f-4a68-84dd-6202c9ba7020 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:43.933755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb63b91c-225e-486e-bede-ceee6404a118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:43.934366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63b91c-225e-486e-bede-ceee6404a118 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.934640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63b91c-225e-486e-bede-ceee6404a118 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.934939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63b91c-225e-486e-bede-ceee6404a118 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.935572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb63b91c-225e-486e-bede-ceee6404a118 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:43.937189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-949b892f-7e7f-4000-9dd3-dd6f84ee1fa6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:43.937513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb63b91c-225e-486e-bede-ceee6404a118 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:43.938390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-949b892f-7e7f-4000-9dd3-dd6f84ee1fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:43.946641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 806/1614] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:43.947458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949b892f-7e7f-4000-9dd3-dd6f84ee1fa6 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:43.950710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-949b892f-7e7f-4000-9dd3-dd6f84ee1fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:43.967830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 809/1615] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:43 2022] POST /identity/v3/projects => generated 405 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:43.969818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668037a-789d-479f-a529-960ca276d7db None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:43.970171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668037a-789d-479f-a529-960ca276d7db None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:43.970846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668037a-789d-479f-a529-960ca276d7db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:43.999583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7668037a-789d-479f-a529-960ca276d7db None None] MFA Rules not processed for user `b34db43f86264c448bf98693bfed7204`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:44.000783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7668037a-789d-479f-a529-960ca276d7db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:44.011191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949b892f-7e7f-4000-9dd3-dd6f84ee1fa6 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.061499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1fa1ff-c36f-42f0-91b2-b3d31c4e40d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.062352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1fa1ff-c36f-42f0-91b2-b3d31c4e40d1 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.062833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1fa1ff-c36f-42f0-91b2-b3d31c4e40d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.063263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1fa1ff-c36f-42f0-91b2-b3d31c4e40d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.064727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 810/1616] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:44.065999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3983419-74bd-4b94-b9f4-46dac1b57c81 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.066229 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3983419-74bd-4b94-b9f4-46dac1b57c81 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.066500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3983419-74bd-4b94-b9f4-46dac1b57c81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.086529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a3983419-74bd-4b94-b9f4-46dac1b57c81 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:44.087695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a3983419-74bd-4b94-b9f4-46dac1b57c81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:44.089708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:44.132187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1fa1ff-c36f-42f0-91b2-b3d31c4e40d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.133670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1fa1ff-c36f-42f0-91b2-b3d31c4e40d1 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.143869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 807/1617] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:44.144740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1fa1ff-c36f-42f0-91b2-b3d31c4e40d1 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.155702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1fa1ff-c36f-42f0-91b2-b3d31c4e40d1 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.160414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c58c21-9e23-4103-86fa-e5e5fd8fb76d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.160979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c58c21-9e23-4103-86fa-e5e5fd8fb76d service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.161267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c58c21-9e23-4103-86fa-e5e5fd8fb76d service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.161603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c58c21-9e23-4103-86fa-e5e5fd8fb76d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.165672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 811/1618] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:40:44 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:44.166347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3983419-74bd-4b94-b9f4-46dac1b57c81 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.171840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c58c21-9e23-4103-86fa-e5e5fd8fb76d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.173318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c58c21-9e23-4103-86fa-e5e5fd8fb76d service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.180927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:44.188374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 808/1619] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:44.189214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c58c21-9e23-4103-86fa-e5e5fd8fb76d service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.198313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c58c21-9e23-4103-86fa-e5e5fd8fb76d service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.211474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe275e6-ed85-4623-a87b-17aeb232b332 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.212072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe275e6-ed85-4623-a87b-17aeb232b332 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.212388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe275e6-ed85-4623-a87b-17aeb232b332 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.212636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe275e6-ed85-4623-a87b-17aeb232b332 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.228274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe275e6-ed85-4623-a87b-17aeb232b332 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.231256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe275e6-ed85-4623-a87b-17aeb232b332 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.235849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3983419-74bd-4b94-b9f4-46dac1b57c81 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.241423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81914a14-888b-42eb-87ec-f2ec04888378 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.242318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81914a14-888b-42eb-87ec-f2ec04888378 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.242641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81914a14-888b-42eb-87ec-f2ec04888378 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.242998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81914a14-888b-42eb-87ec-f2ec04888378 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.246640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 809/1620] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:44.247895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe275e6-ed85-4623-a87b-17aeb232b332 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.262634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe275e6-ed85-4623-a87b-17aeb232b332 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.269693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b352f5-8948-424c-9df3-59ef06713164 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.270947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b352f5-8948-424c-9df3-59ef06713164 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.270947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b352f5-8948-424c-9df3-59ef06713164 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.271308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b352f5-8948-424c-9df3-59ef06713164 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.289485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b352f5-8948-424c-9df3-59ef06713164 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.291680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b352f5-8948-424c-9df3-59ef06713164 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.306981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 810/1621] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:44.307942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b352f5-8948-424c-9df3-59ef06713164 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.322465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b352f5-8948-424c-9df3-59ef06713164 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.328669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441800ec-3c55-4a1d-8919-57d0f3ab095e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.329280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441800ec-3c55-4a1d-8919-57d0f3ab095e admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.329558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441800ec-3c55-4a1d-8919-57d0f3ab095e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.329830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441800ec-3c55-4a1d-8919-57d0f3ab095e admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.330499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441800ec-3c55-4a1d-8919-57d0f3ab095e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.332822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441800ec-3c55-4a1d-8919-57d0f3ab095e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.339873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81914a14-888b-42eb-87ec-f2ec04888378 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.341906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81914a14-888b-42eb-87ec-f2ec04888378 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.356567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 812/1622] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:44 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:44.358005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81914a14-888b-42eb-87ec-f2ec04888378 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.370966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81914a14-888b-42eb-87ec-f2ec04888378 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.377410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c93715-8d1b-4012-ac9e-cbe6ad3f21cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.378020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 811/1623] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:44 2022] POST /identity/v3/users => generated 309 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:44.378122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c93715-8d1b-4012-ac9e-cbe6ad3f21cf admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.378550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c93715-8d1b-4012-ac9e-cbe6ad3f21cf admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.379428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441800ec-3c55-4a1d-8919-57d0f3ab095e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.382428 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c93715-8d1b-4012-ac9e-cbe6ad3f21cf admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.384286 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c93715-8d1b-4012-ac9e-cbe6ad3f21cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.386843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c93715-8d1b-4012-ac9e-cbe6ad3f21cf admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.428384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 813/1624] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:44 2022] POST /identity/v3/projects => generated 371 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:44.428384 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c93715-8d1b-4012-ac9e-cbe6ad3f21cf admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.431323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441800ec-3c55-4a1d-8919-57d0f3ab095e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.519134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c93715-8d1b-4012-ac9e-cbe6ad3f21cf admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.525654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b4b4f8-7863-4327-924d-634b897e8717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.525958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4b4f8-7863-4327-924d-634b897e8717 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.526462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4b4f8-7863-4327-924d-634b897e8717 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.526918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4b4f8-7863-4327-924d-634b897e8717 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.527586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b4b4f8-7863-4327-924d-634b897e8717 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.530401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b4b4f8-7863-4327-924d-634b897e8717 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.544985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 814/1625] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:44.550692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b4b4f8-7863-4327-924d-634b897e8717 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.559808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193680e1-ebd0-457f-815d-301c0f25596c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.560431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193680e1-ebd0-457f-815d-301c0f25596c service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.560711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193680e1-ebd0-457f-815d-301c0f25596c service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.560980 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193680e1-ebd0-457f-815d-301c0f25596c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.565908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b4b4f8-7863-4327-924d-634b897e8717 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.573088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847cac3c-337d-4584-806c-dfe156fcbc08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.573777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847cac3c-337d-4584-806c-dfe156fcbc08 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.574212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847cac3c-337d-4584-806c-dfe156fcbc08 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.574666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847cac3c-337d-4584-806c-dfe156fcbc08 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.576020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847cac3c-337d-4584-806c-dfe156fcbc08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.578951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847cac3c-337d-4584-806c-dfe156fcbc08 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.626647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 815/1626] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:44 2022] POST /identity/v3/users => generated 292 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:44.627478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847cac3c-337d-4584-806c-dfe156fcbc08 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.651703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847cac3c-337d-4584-806c-dfe156fcbc08 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.667018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09905e12-ccf0-485d-8cda-482701801a0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.671939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09905e12-ccf0-485d-8cda-482701801a0e admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.672406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09905e12-ccf0-485d-8cda-482701801a0e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.672692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09905e12-ccf0-485d-8cda-482701801a0e admin admin] PATH_INFO: `/v3/projects/d9ee185eabba49af95c5e49aa541d42a/users/7266059591674b988578668ebc245e4a/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.717486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09905e12-ccf0-485d-8cda-482701801a0e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9ee185eabba49af95c5e49aa541d42a, user_id=7266059591674b988578668ebc245e4a, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.723446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09905e12-ccf0-485d-8cda-482701801a0e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.739457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193680e1-ebd0-457f-815d-301c0f25596c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.740940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193680e1-ebd0-457f-815d-301c0f25596c service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.763850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:44.770793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 816/1627] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:44 2022] PUT /identity/v3/projects/d9ee185eabba49af95c5e49aa541d42a/users/7266059591674b988578668ebc245e4a/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:44.774380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09905e12-ccf0-485d-8cda-482701801a0e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.775004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 812/1628] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:44.778467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193680e1-ebd0-457f-815d-301c0f25596c service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.795178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:44.806755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193680e1-ebd0-457f-815d-301c0f25596c service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.929600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09905e12-ccf0-485d-8cda-482701801a0e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:44.947903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db9f27-0c7d-43f8-9670-4c18367e3a58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.948667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db9f27-0c7d-43f8-9670-4c18367e3a58 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.949490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db9f27-0c7d-43f8-9670-4c18367e3a58 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.949490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db9f27-0c7d-43f8-9670-4c18367e3a58 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.950461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700d2acf-24ad-4ace-b9e2-33213fefde2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:44.950629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58db9f27-0c7d-43f8-9670-4c18367e3a58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.951362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d2acf-24ad-4ace-b9e2-33213fefde2d admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:44.951964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d2acf-24ad-4ace-b9e2-33213fefde2d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:44.952246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58db9f27-0c7d-43f8-9670-4c18367e3a58 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.953190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d2acf-24ad-4ace-b9e2-33213fefde2d admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:44.955204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700d2acf-24ad-4ace-b9e2-33213fefde2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:44.957276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700d2acf-24ad-4ace-b9e2-33213fefde2d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:44.970787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 817/1629] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:44.975792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db9f27-0c7d-43f8-9670-4c18367e3a58 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:44.990643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 813/1630] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:44.995534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700d2acf-24ad-4ace-b9e2-33213fefde2d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:45.007951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db9f27-0c7d-43f8-9670-4c18367e3a58 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:45.022155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902d4742-773c-4537-8a86-1a01a2331a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:45.022907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902d4742-773c-4537-8a86-1a01a2331a27 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:45.023196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902d4742-773c-4537-8a86-1a01a2331a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:45.030505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902d4742-773c-4537-8a86-1a01a2331a27 admin admin] PATH_INFO: `/v3/projects/9152791b9472426d91b410ab314aeb64/users/277e24c143ca427eaae9c9f9caf79c50/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:45.035314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700d2acf-24ad-4ace-b9e2-33213fefde2d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:45.063250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902d4742-773c-4537-8a86-1a01a2331a27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9152791b9472426d91b410ab314aeb64, user_id=277e24c143ca427eaae9c9f9caf79c50, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:45.064948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220409f7-a287-4f4a-b6f1-a3812da54ccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:45.066345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220409f7-a287-4f4a-b6f1-a3812da54ccc admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:45.066619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902d4742-773c-4537-8a86-1a01a2331a27 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:45.070711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220409f7-a287-4f4a-b6f1-a3812da54ccc admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:45.071227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220409f7-a287-4f4a-b6f1-a3812da54ccc admin admin] PATH_INFO: `/v3/projects/d9ee185eabba49af95c5e49aa541d42a/users/7266059591674b988578668ebc245e4a/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:45.087993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220409f7-a287-4f4a-b6f1-a3812da54ccc admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9ee185eabba49af95c5e49aa541d42a, user_id=7266059591674b988578668ebc245e4a, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:45.096024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220409f7-a287-4f4a-b6f1-a3812da54ccc admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:45.126170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 818/1631] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:44 2022] PUT /identity/v3/projects/9152791b9472426d91b410ab314aeb64/users/277e24c143ca427eaae9c9f9caf79c50/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:45.132765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902d4742-773c-4537-8a86-1a01a2331a27 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:45.164746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 814/1632] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:44 2022] PUT /identity/v3/projects/d9ee185eabba49af95c5e49aa541d42a/users/7266059591674b988578668ebc245e4a/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:40:45.167169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f93aa-d070-4449-a4ab-871aad39c6de None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:45.167655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f93aa-d070-4449-a4ab-871aad39c6de None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:45.168127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f93aa-d070-4449-a4ab-871aad39c6de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:45.247290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902d4742-773c-4537-8a86-1a01a2331a27 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:45.261443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-092f93aa-d070-4449-a4ab-871aad39c6de None None] MFA Rules not processed for user `7266059591674b988578668ebc245e4a`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:45.267073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-092f93aa-d070-4449-a4ab-871aad39c6de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:45.322251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a69ee6b-4f16-4526-b991-363cc4dce0aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:45.323103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69ee6b-4f16-4526-b991-363cc4dce0aa admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:45.323445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69ee6b-4f16-4526-b991-363cc4dce0aa admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:45.323759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69ee6b-4f16-4526-b991-363cc4dce0aa admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:45.324303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a69ee6b-4f16-4526-b991-363cc4dce0aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:45.332383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a69ee6b-4f16-4526-b991-363cc4dce0aa admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:45.364375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 819/1633] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:45.367418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a69ee6b-4f16-4526-b991-363cc4dce0aa admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:45.386059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a69ee6b-4f16-4526-b991-363cc4dce0aa admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:45.399999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29b78bb-befb-4758-abcc-532577babc2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:45.400680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29b78bb-befb-4758-abcc-532577babc2a admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:45.401134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29b78bb-befb-4758-abcc-532577babc2a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:45.401285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29b78bb-befb-4758-abcc-532577babc2a admin admin] PATH_INFO: `/v3/projects/9152791b9472426d91b410ab314aeb64/users/277e24c143ca427eaae9c9f9caf79c50/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:45.404624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b29b78bb-befb-4758-abcc-532577babc2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9152791b9472426d91b410ab314aeb64, user_id=277e24c143ca427eaae9c9f9caf79c50, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:45.407464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b29b78bb-befb-4758-abcc-532577babc2a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:45.451186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 820/1634] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:45 2022] PUT /identity/v3/projects/9152791b9472426d91b410ab314aeb64/users/277e24c143ca427eaae9c9f9caf79c50/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:40:45.457898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654df67b-ebb3-49de-a91b-66700d246951 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:45.458341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654df67b-ebb3-49de-a91b-66700d246951 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:45.458696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654df67b-ebb3-49de-a91b-66700d246951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:45.468761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:45.533945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-654df67b-ebb3-49de-a91b-66700d246951 None None] MFA Rules not processed for user `277e24c143ca427eaae9c9f9caf79c50`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:45.535263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-654df67b-ebb3-49de-a91b-66700d246951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:45.589310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 815/1635] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:45 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 424 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:45.667811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092f93aa-d070-4449-a4ab-871aad39c6de None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:45.751667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 821/1636] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:45 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:40:45.859850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092f93aa-d070-4449-a4ab-871aad39c6de None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:45.907965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:45.965535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3d231a-ad68-4634-876f-fb1f0d4df7c2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:45.966447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d231a-ad68-4634-876f-fb1f0d4df7c2 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:45.967514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d231a-ad68-4634-876f-fb1f0d4df7c2 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:45.967947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d231a-ad68-4634-876f-fb1f0d4df7c2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:46.100807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654df67b-ebb3-49de-a91b-66700d246951 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:46.113179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654df67b-ebb3-49de-a91b-66700d246951 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:46.123560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba64190c-df0b-44b5-9e26-7b87910e6f2c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:46.124294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64190c-df0b-44b5-9e26-7b87910e6f2c service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:46.126775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64190c-df0b-44b5-9e26-7b87910e6f2c service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:46.128994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64190c-df0b-44b5-9e26-7b87910e6f2c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:46.132407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3d231a-ad68-4634-876f-fb1f0d4df7c2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:46.135695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3d231a-ad68-4634-876f-fb1f0d4df7c2 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:46.149487 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba64190c-df0b-44b5-9e26-7b87910e6f2c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:46.152472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba64190c-df0b-44b5-9e26-7b87910e6f2c service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:46.164020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 816/1637] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:45 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:46.186707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 822/1638] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:46 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:40:46.373469 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3d231a-ad68-4634-876f-fb1f0d4df7c2 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:46.387582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3d231a-ad68-4634-876f-fb1f0d4df7c2 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:46.394774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f1b97-7f73-44f6-8dfe-a37324af7d15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:46.395908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f1b97-7f73-44f6-8dfe-a37324af7d15 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:46.396210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f1b97-7f73-44f6-8dfe-a37324af7d15 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:46.398529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f1b97-7f73-44f6-8dfe-a37324af7d15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:46.540725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba64190c-df0b-44b5-9e26-7b87910e6f2c service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:46.612730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64f1b97-7f73-44f6-8dfe-a37324af7d15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:46.616820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64f1b97-7f73-44f6-8dfe-a37324af7d15 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:46.640959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 817/1639] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:46.686063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba64190c-df0b-44b5-9e26-7b87910e6f2c service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:46.693049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:46.693702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:46.694043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:46.694321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] PATH_INFO: `/v3/users/ad16cd3ff11e41428db531180e088338` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:46.695523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad16cd3ff11e41428db531180e088338)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:46.696600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: WARNING py.warnings [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:40:46.696600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: warnings.warn(msg) Aug 29 22:40:46.696600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: Aug 29 22:40:46.698290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:46.716513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213c6799-6ea3-4512-ba7f-c6a952d6d4b5 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:46.716838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213c6799-6ea3-4512-ba7f-c6a952d6d4b5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:46.717073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213c6799-6ea3-4512-ba7f-c6a952d6d4b5 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:46.719686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 818/1640] 10.222.0.77 () {54 vars in 962 bytes} [Mon Aug 29 22:40:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:40:46.735691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc18a9-abe4-4aa1-9acd-39e6db5aa891 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:46.736123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc18a9-abe4-4aa1-9acd-39e6db5aa891 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:46.736478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc18a9-abe4-4aa1-9acd-39e6db5aa891 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:46.775998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-45cc18a9-abe4-4aa1-9acd-39e6db5aa891 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:46.777418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-45cc18a9-abe4-4aa1-9acd-39e6db5aa891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:46.861400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad16cd3ff11e41428db531180e088338'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:46.864161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad16cd3ff11e41428db531180e088338'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:46.890603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad16cd3ff11e41428db531180e088338'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:46.928329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad16cd3ff11e41428db531180e088338'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:46.967240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 819/1641] 10.222.0.77 () {58 vars in 1059 bytes} [Mon Aug 29 22:40:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:46.970639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:46.973734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-813c92b3-5de2-44e4-b242-d0bbcc5ee3b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad16cd3ff11e41428db531180e088338'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:46.978199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92f4b4-b578-419c-a295-4ac6fdc086fe None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:46.978693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92f4b4-b578-419c-a295-4ac6fdc086fe None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:46.979234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92f4b4-b578-419c-a295-4ac6fdc086fe None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:46.982423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 820/1642] 10.222.0.77 () {54 vars in 912 bytes} [Mon Aug 29 22:40:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:40:46.990691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2de28d7-c6b5-4a56-936f-04cc57b0bab9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:46.991239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2de28d7-c6b5-4a56-936f-04cc57b0bab9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:46.992041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2de28d7-c6b5-4a56-936f-04cc57b0bab9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:47.003625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 823/1643] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:40:46 2022] DELETE /identity/v3/users/ad16cd3ff11e41428db531180e088338 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:47.008464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7924ea33-5d54-4479-9f94-3ff4581e510c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:47.008756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7924ea33-5d54-4479-9f94-3ff4581e510c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:47.010192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7924ea33-5d54-4479-9f94-3ff4581e510c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:47.021263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e2de28d7-c6b5-4a56-936f-04cc57b0bab9 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:47.022912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e2de28d7-c6b5-4a56-936f-04cc57b0bab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:47.030368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:47.067210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7924ea33-5d54-4479-9f94-3ff4581e510c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:47.068294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7924ea33-5d54-4479-9f94-3ff4581e510c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:47.143865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 821/1644] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:40:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:47.159466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2de28d7-c6b5-4a56-936f-04cc57b0bab9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:47.161485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e2de28d7-c6b5-4a56-936f-04cc57b0bab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:47.210618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 824/1645] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:47.232091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7924ea33-5d54-4479-9f94-3ff4581e510c None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:47.234127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7924ea33-5d54-4479-9f94-3ff4581e510c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:47.277156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2de28d7-c6b5-4a56-936f-04cc57b0bab9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:47.281720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:47.282508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:47.282924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:47.283298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:47.297037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:47.298567 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:47.315388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 822/1646] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:47.393162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7924ea33-5d54-4479-9f94-3ff4581e510c None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:47.401791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1e31dc-ff22-43e5-94e7-398c401fdda3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:47.403295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e31dc-ff22-43e5-94e7-398c401fdda3 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:47.404583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e31dc-ff22-43e5-94e7-398c401fdda3 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:47.407053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e31dc-ff22-43e5-94e7-398c401fdda3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:47.431963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1e31dc-ff22-43e5-94e7-398c401fdda3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:47.439383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1e31dc-ff22-43e5-94e7-398c401fdda3 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:47.463800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 825/1647] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:40:48.711090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:48.713314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:48.734810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c0d96-2b17-4196-8e28-4a5537da16eb service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:48.766815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:48.767664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:48.768113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:48.769308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] PATH_INFO: `/v3/projects/b847982223024b6dadd05dde4b7a3b29` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:48.775962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b847982223024b6dadd05dde4b7a3b29)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:48.777020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: WARNING py.warnings [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 29 22:40:48.777020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: warnings.warn(msg) Aug 29 22:40:48.777020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: Aug 29 22:40:48.778441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:48.935414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b847982223024b6dadd05dde4b7a3b29', 'request_id': 'req-05112da4-4231-4b6b-a449-f95599bed07e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:48.966795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b847982223024b6dadd05dde4b7a3b29', 'request_id': 'req-05112da4-4231-4b6b-a449-f95599bed07e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:48.982123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b847982223024b6dadd05dde4b7a3b29', 'request_id': 'req-05112da4-4231-4b6b-a449-f95599bed07e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:49.023405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] The token cache is being invalidate because project b847982223024b6dadd05dde4b7a3b29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:40:49.023705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:49.035924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:49.049545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 823/1648] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:40:48 2022] DELETE /identity/v3/projects/b847982223024b6dadd05dde4b7a3b29 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:49.054983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1e31dc-ff22-43e5-94e7-398c401fdda3 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:49.089612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1e31dc-ff22-43e5-94e7-398c401fdda3 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:49.274134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:49.275050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:49.275297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:49.275571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] PATH_INFO: `/v3/users/7266059591674b988578668ebc245e4a` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:49.292600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7266059591674b988578668ebc245e4a)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:49.299197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:49.479435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=6,keystone:SELECT=94 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:49.506445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7266059591674b988578668ebc245e4a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:49.509284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7266059591674b988578668ebc245e4a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:49.530263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7266059591674b988578668ebc245e4a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:49.546954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7266059591674b988578668ebc245e4a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:49.589245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-979eaae3-9afa-4338-bc73-7ea887387ae6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7266059591674b988578668ebc245e4a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:49.604292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 826/1649] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:40:49 2022] DELETE /identity/v3/users/7266059591674b988578668ebc245e4a => generated 0 bytes in 550 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:51.315569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:51.319368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:51.374603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05112da4-4231-4b6b-a449-f95599bed07e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:51.460663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:51.589832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:51.592628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:51.593819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:51.595649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] PATH_INFO: `/v3/projects/d9ee185eabba49af95c5e49aa541d42a` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:51.599395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9ee185eabba49af95c5e49aa541d42a)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:51.604665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:51.733871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9ee185eabba49af95c5e49aa541d42a', 'request_id': 'req-d02bea8d-23fb-43ec-8781-e557e9be94af'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:51.748116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9ee185eabba49af95c5e49aa541d42a', 'request_id': 'req-d02bea8d-23fb-43ec-8781-e557e9be94af'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:51.767029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9ee185eabba49af95c5e49aa541d42a', 'request_id': 'req-d02bea8d-23fb-43ec-8781-e557e9be94af'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:51.791276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] The token cache is being invalidate because project d9ee185eabba49af95c5e49aa541d42a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:40:51.791691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:51.795751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d02bea8d-23fb-43ec-8781-e557e9be94af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:51.802213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 824/1650] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:40:51 2022] DELETE /identity/v3/projects/d9ee185eabba49af95c5e49aa541d42a => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:51.813070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f78d53-2ffc-4440-95db-0e8e7e7d25be None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:51.813632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f78d53-2ffc-4440-95db-0e8e7e7d25be None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:51.813632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f78d53-2ffc-4440-95db-0e8e7e7d25be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:51.889721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a4f78d53-2ffc-4440-95db-0e8e7e7d25be None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:51.896933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a4f78d53-2ffc-4440-95db-0e8e7e7d25be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:52.078279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed4e380-3f6f-46f5-9dc8-f023aa4b4f8d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:52.078590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed4e380-3f6f-46f5-9dc8-f023aa4b4f8d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:52.078869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed4e380-3f6f-46f5-9dc8-f023aa4b4f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:52.082250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 827/1651] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:52.143362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3ed4e380-3f6f-46f5-9dc8-f023aa4b4f8d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:52.146665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3ed4e380-3f6f-46f5-9dc8-f023aa4b4f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:52.166984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 825/1652] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:52.173554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f78d53-2ffc-4440-95db-0e8e7e7d25be None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:52.200996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f78d53-2ffc-4440-95db-0e8e7e7d25be None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:52.212202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:52.212966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:52.213449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:52.214641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:52.214944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:40:52.215404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:52.220516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:52.256264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 828/1653] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:40:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:40:52.268256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed4e380-3f6f-46f5-9dc8-f023aa4b4f8d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:52.285241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed4e380-3f6f-46f5-9dc8-f023aa4b4f8d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:52.331763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c0137-46aa-4e84-a9d8-dd650b1c1e05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:52.333114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c0137-46aa-4e84-a9d8-dd650b1c1e05 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:52.333529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c0137-46aa-4e84-a9d8-dd650b1c1e05 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:52.333926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c0137-46aa-4e84-a9d8-dd650b1c1e05 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:52.334787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498c0137-46aa-4e84-a9d8-dd650b1c1e05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:52.336869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498c0137-46aa-4e84-a9d8-dd650b1c1e05 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:52.435032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:52.440303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 826/1654] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:52 2022] POST /identity/v3/projects => generated 373 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:52.446055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca08e4-7a7e-4b9a-9f3b-6fd51c75793c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:52.638544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47807ed2-b254-4a4e-a036-5c4d5c1d9717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:52.639202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47807ed2-b254-4a4e-a036-5c4d5c1d9717 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:52.639478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47807ed2-b254-4a4e-a036-5c4d5c1d9717 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:52.639745 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47807ed2-b254-4a4e-a036-5c4d5c1d9717 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:52.646571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47807ed2-b254-4a4e-a036-5c4d5c1d9717 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:52.648607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47807ed2-b254-4a4e-a036-5c4d5c1d9717 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:52.711355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c0137-46aa-4e84-a9d8-dd650b1c1e05 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:52.711775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 829/1655] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:52 2022] POST /identity/v3/users => generated 287 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:40:52.733953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c0137-46aa-4e84-a9d8-dd650b1c1e05 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:52.746787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ae3a1-5675-44af-b799-5420f7c68aee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:52.747459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ae3a1-5675-44af-b799-5420f7c68aee admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:52.747737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ae3a1-5675-44af-b799-5420f7c68aee admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:52.748005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ae3a1-5675-44af-b799-5420f7c68aee admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:52.748667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60ae3a1-5675-44af-b799-5420f7c68aee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:52.750871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60ae3a1-5675-44af-b799-5420f7c68aee admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:52.825121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 827/1656] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:52 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:40:52.829497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47807ed2-b254-4a4e-a036-5c4d5c1d9717 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:52.839874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47807ed2-b254-4a4e-a036-5c4d5c1d9717 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:52.885182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49e6473-0a43-446d-a457-32cf7d62f5c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:52.893759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49e6473-0a43-446d-a457-32cf7d62f5c2 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:52.894042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49e6473-0a43-446d-a457-32cf7d62f5c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:52.894528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49e6473-0a43-446d-a457-32cf7d62f5c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:52.895519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49e6473-0a43-446d-a457-32cf7d62f5c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:52.901732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49e6473-0a43-446d-a457-32cf7d62f5c2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:52.968650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ae3a1-5675-44af-b799-5420f7c68aee admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:52.970864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 830/1657] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:52 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:52.991750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ae3a1-5675-44af-b799-5420f7c68aee admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:53.010881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcdee6b-368a-475b-a61f-c71bcdb976d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:53.011561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcdee6b-368a-475b-a61f-c71bcdb976d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:53.012029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcdee6b-368a-475b-a61f-c71bcdb976d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:53.012446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcdee6b-368a-475b-a61f-c71bcdb976d2 admin admin] PATH_INFO: `/v3/projects/36c72eadb72f43f9ab79dae5bc3ce054/users/17af7063865245fea3c139ddecd6874c/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:53.055625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcdee6b-368a-475b-a61f-c71bcdb976d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36c72eadb72f43f9ab79dae5bc3ce054, user_id=17af7063865245fea3c139ddecd6874c, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:53.068015 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcdee6b-368a-475b-a61f-c71bcdb976d2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:53.102687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 828/1658] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:52 2022] PUT /identity/v3/projects/36c72eadb72f43f9ab79dae5bc3ce054/users/17af7063865245fea3c139ddecd6874c/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:40:53.113388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209924c-71a3-4524-b6b7-9410bff90395 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:53.113669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209924c-71a3-4524-b6b7-9410bff90395 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:53.113940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209924c-71a3-4524-b6b7-9410bff90395 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:53.201564 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3209924c-71a3-4524-b6b7-9410bff90395 None None] MFA Rules not processed for user `17af7063865245fea3c139ddecd6874c`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:53.204429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3209924c-71a3-4524-b6b7-9410bff90395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:53.239626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:53.410457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcdee6b-368a-475b-a61f-c71bcdb976d2 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:53.423919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 831/1659] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:53.598488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:53.620240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcdee6b-368a-475b-a61f-c71bcdb976d2 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:53.638209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c47b4c2-a0d6-4350-ab7a-bf25d92ec6b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:53.638887 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47b4c2-a0d6-4350-ab7a-bf25d92ec6b2 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:53.639175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47b4c2-a0d6-4350-ab7a-bf25d92ec6b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:53.639452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47b4c2-a0d6-4350-ab7a-bf25d92ec6b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:53.640141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c47b4c2-a0d6-4350-ab7a-bf25d92ec6b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:53.647308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c47b4c2-a0d6-4350-ab7a-bf25d92ec6b2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:53.707782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 829/1660] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:40:53 2022] POST /identity/v3/projects => generated 373 bytes in 297 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:40:53.710094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3209924c-71a3-4524-b6b7-9410bff90395 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:53.730187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3209924c-71a3-4524-b6b7-9410bff90395 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:53.914228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b6238e-6232-4e47-b3cc-e4280e2c9223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:53.914885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b6238e-6232-4e47-b3cc-e4280e2c9223 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:53.915417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b6238e-6232-4e47-b3cc-e4280e2c9223 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:53.915827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b6238e-6232-4e47-b3cc-e4280e2c9223 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:53.916627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b6238e-6232-4e47-b3cc-e4280e2c9223 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:53.918803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b6238e-6232-4e47-b3cc-e4280e2c9223 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:53.988400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 832/1661] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:40:53 2022] POST /identity/v3/users => generated 293 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:40:53.995191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c47b4c2-a0d6-4350-ab7a-bf25d92ec6b2 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:54.023798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c47b4c2-a0d6-4350-ab7a-bf25d92ec6b2 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:54.039650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a0eb6b-5f51-4e9b-bec7-3168e1191abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:54.040286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a0eb6b-5f51-4e9b-bec7-3168e1191abc admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:54.040562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a0eb6b-5f51-4e9b-bec7-3168e1191abc admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:54.040828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a0eb6b-5f51-4e9b-bec7-3168e1191abc admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:54.041488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a0eb6b-5f51-4e9b-bec7-3168e1191abc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:54.043281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a0eb6b-5f51-4e9b-bec7-3168e1191abc admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:54.065947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 830/1662] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:53 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:54.066338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b6238e-6232-4e47-b3cc-e4280e2c9223 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:54.090625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b6238e-6232-4e47-b3cc-e4280e2c9223 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:54.099632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb533c-0df2-4ac7-bedc-583db64eca83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:54.100685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb533c-0df2-4ac7-bedc-583db64eca83 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:54.101109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb533c-0df2-4ac7-bedc-583db64eca83 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:54.102282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb533c-0df2-4ac7-bedc-583db64eca83 admin admin] PATH_INFO: `/v3/projects/7b6cabe35cad479ba16eb542a8972313/users/ae05283ab16e437f82eb00fd283d4057/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:54.129307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eb533c-0df2-4ac7-bedc-583db64eca83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b6cabe35cad479ba16eb542a8972313, user_id=ae05283ab16e437f82eb00fd283d4057, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:54.132311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eb533c-0df2-4ac7-bedc-583db64eca83 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:54.155710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 833/1663] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:54 2022] PUT /identity/v3/projects/7b6cabe35cad479ba16eb542a8972313/users/ae05283ab16e437f82eb00fd283d4057/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:40:54.160441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a0eb6b-5f51-4e9b-bec7-3168e1191abc admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:54.176119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a0eb6b-5f51-4e9b-bec7-3168e1191abc admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:54.382152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5772f80-de73-44fc-8f69-83b12d1c20fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:54.383191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5772f80-de73-44fc-8f69-83b12d1c20fa admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:54.384332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5772f80-de73-44fc-8f69-83b12d1c20fa admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:54.384708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5772f80-de73-44fc-8f69-83b12d1c20fa admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:54.385753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5772f80-de73-44fc-8f69-83b12d1c20fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:54.387558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5772f80-de73-44fc-8f69-83b12d1c20fa admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:54.414261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 831/1664] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:40:54 2022] GET /identity/v3/roles => generated 1420 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:40:54.418249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb533c-0df2-4ac7-bedc-583db64eca83 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:54.435412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb533c-0df2-4ac7-bedc-583db64eca83 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:54.455709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:54.457141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:54.457512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:54.457791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] PATH_INFO: `/v3/projects/7b6cabe35cad479ba16eb542a8972313/users/ae05283ab16e437f82eb00fd283d4057/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:54.462473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b6cabe35cad479ba16eb542a8972313, user_id=ae05283ab16e437f82eb00fd283d4057, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:54.476002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:54.500648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 834/1665] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:40:54 2022] PUT /identity/v3/projects/7b6cabe35cad479ba16eb542a8972313/users/ae05283ab16e437f82eb00fd283d4057/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:54.519796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1cbfea-beeb-420b-853f-04adeb166987 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:54.520383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1cbfea-beeb-420b-853f-04adeb166987 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:54.520876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1cbfea-beeb-420b-853f-04adeb166987 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:54.639674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8c1cbfea-beeb-420b-853f-04adeb166987 None None] MFA Rules not processed for user `ae05283ab16e437f82eb00fd283d4057`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:54.641636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8c1cbfea-beeb-420b-853f-04adeb166987 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:54.948291 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 832/1666] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:54 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 431 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:54.956999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:54.959228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:54.987297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b6967-92cd-44f8-ac45-9afbdbd7a3fa None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:54.987650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b6967-92cd-44f8-ac45-9afbdbd7a3fa None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:54.988097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b6967-92cd-44f8-ac45-9afbdbd7a3fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:55.043874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-819b6967-92cd-44f8-ac45-9afbdbd7a3fa None None] MFA Rules not processed for user `72c45a0b5705455c8bb3f09053e325cf`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:55.045112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-819b6967-92cd-44f8-ac45-9afbdbd7a3fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:55.086867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0d5b47-eb1d-4a7d-9572-5642ee1f4b70 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:55.091473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c423cba7-4992-4afd-a468-a7d7939fd3d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:55.092120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c423cba7-4992-4afd-a468-a7d7939fd3d2 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:55.092398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c423cba7-4992-4afd-a468-a7d7939fd3d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:55.092673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c423cba7-4992-4afd-a468-a7d7939fd3d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:55.096936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:55.112340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c423cba7-4992-4afd-a468-a7d7939fd3d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:55.113805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c423cba7-4992-4afd-a468-a7d7939fd3d2 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:55.138803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 835/1667] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:54 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:55.141595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 833/1668] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:55.155206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073847b0-ddba-47d4-a572-5c0280079ce2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:55.156085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073847b0-ddba-47d4-a572-5c0280079ce2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:55.156533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073847b0-ddba-47d4-a572-5c0280079ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:55.170894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:55.181432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-073847b0-ddba-47d4-a572-5c0280079ce2 None None] MFA Rules not processed for user `72c45a0b5705455c8bb3f09053e325cf`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:55.185289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-073847b0-ddba-47d4-a572-5c0280079ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:55.210338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 836/1669] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:55 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:55.223860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b6967-92cd-44f8-ac45-9afbdbd7a3fa None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:55.225543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-819b6967-92cd-44f8-ac45-9afbdbd7a3fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:55.360959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b6967-92cd-44f8-ac45-9afbdbd7a3fa None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:55.369391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a9a82c-a075-4710-89fc-f6951979a447 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:55.370826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a9a82c-a075-4710-89fc-f6951979a447 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:55.370826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a9a82c-a075-4710-89fc-f6951979a447 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:55.370826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a9a82c-a075-4710-89fc-f6951979a447 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:55.380419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a9a82c-a075-4710-89fc-f6951979a447 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:55.382206 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a9a82c-a075-4710-89fc-f6951979a447 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:55.394334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 834/1670] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:55 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:55.820801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073847b0-ddba-47d4-a572-5c0280079ce2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:55.948568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073847b0-ddba-47d4-a572-5c0280079ce2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:55.961461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:55.962908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:55.963292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:55.963669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] PATH_INFO: `/v3/users/17af7063865245fea3c139ddecd6874c` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:55.965210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] RBAC: Authorizing `identity:delete_user(user_id=17af7063865245fea3c139ddecd6874c)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:55.969328 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:56.068457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17af7063865245fea3c139ddecd6874c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:56.073319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17af7063865245fea3c139ddecd6874c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:56.086963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17af7063865245fea3c139ddecd6874c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:56.103988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17af7063865245fea3c139ddecd6874c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:56.133393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17af7063865245fea3c139ddecd6874c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:56.148071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 837/1671] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:40:55 2022] DELETE /identity/v3/users/17af7063865245fea3c139ddecd6874c => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:56.152233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ce1acb-1955-45d7-8b3b-83c496c58ec2 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:56.153124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ce1acb-1955-45d7-8b3b-83c496c58ec2 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:56.153601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ce1acb-1955-45d7-8b3b-83c496c58ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:56.193859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d8ce1acb-1955-45d7-8b3b-83c496c58ec2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:56.196109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d8ce1acb-1955-45d7-8b3b-83c496c58ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:56.307253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 835/1672] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:56.341724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:56.343346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:56.447041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d3c2ab-5f5e-4151-bf84-806b13c1d144 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:56.453623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce7b689-7095-4f56-abd1-908fcdf885dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:56.454585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce7b689-7095-4f56-abd1-908fcdf885dd service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:56.455000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce7b689-7095-4f56-abd1-908fcdf885dd service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:56.455705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce7b689-7095-4f56-abd1-908fcdf885dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:56.471109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce7b689-7095-4f56-abd1-908fcdf885dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:56.473228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce7b689-7095-4f56-abd1-908fcdf885dd service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:56.488024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 838/1673] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:56.569047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94454a26-f65d-4466-8e9b-03e2d954cc5a None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:56.569383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94454a26-f65d-4466-8e9b-03e2d954cc5a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:56.569705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94454a26-f65d-4466-8e9b-03e2d954cc5a None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:56.571910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 836/1674] 10.222.0.77 () {54 vars in 912 bytes} [Mon Aug 29 22:40:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:40:56.578571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f34b91-67f2-48c6-85f0-5ef17829a7ff None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:56.579075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f34b91-67f2-48c6-85f0-5ef17829a7ff None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:56.579353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f34b91-67f2-48c6-85f0-5ef17829a7ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:56.723607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e9f34b91-67f2-48c6-85f0-5ef17829a7ff None None] MFA Rules not processed for user `dea13198ef8e45de85b92e79e3ee71b0`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:56.728016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e9f34b91-67f2-48c6-85f0-5ef17829a7ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:56.755548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 839/1675] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:40:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:56.798266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94454a26-f65d-4466-8e9b-03e2d954cc5a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:56.804186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-94454a26-f65d-4466-8e9b-03e2d954cc5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:56.844013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94454a26-f65d-4466-8e9b-03e2d954cc5a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:56.857182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea0ff72-6621-4134-9914-a1403b398ed2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:56.857826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea0ff72-6621-4134-9914-a1403b398ed2 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:56.858104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea0ff72-6621-4134-9914-a1403b398ed2 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:56.858401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea0ff72-6621-4134-9914-a1403b398ed2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:56.877559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea0ff72-6621-4134-9914-a1403b398ed2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:56.879054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea0ff72-6621-4134-9914-a1403b398ed2 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:56.904482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 837/1676] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:40:57.147939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f34b91-67f2-48c6-85f0-5ef17829a7ff None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:57.149523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e9f34b91-67f2-48c6-85f0-5ef17829a7ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:57.174653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:57.196403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f34b91-67f2-48c6-85f0-5ef17829a7ff None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:57.213920 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:57.214921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:57.215402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:57.219150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] PATH_INFO: `/v3/users/298bfaf3cc0d478093d3ee97a3c1e734` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:57.222538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=298bfaf3cc0d478093d3ee97a3c1e734)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:57.225245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:57.371783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '298bfaf3cc0d478093d3ee97a3c1e734'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:57.372613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '298bfaf3cc0d478093d3ee97a3c1e734'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:57.390406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '298bfaf3cc0d478093d3ee97a3c1e734'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:57.407931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '298bfaf3cc0d478093d3ee97a3c1e734'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:57.446602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '298bfaf3cc0d478093d3ee97a3c1e734'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:57.479982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 840/1677] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:40:57 2022] DELETE /identity/v3/users/298bfaf3cc0d478093d3ee97a3c1e734 => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:57.486109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd04822-c7db-47d2-b698-618548f5162d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:57.487586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd04822-c7db-47d2-b698-618548f5162d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:57.489082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd04822-c7db-47d2-b698-618548f5162d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:57.542306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8fd04822-c7db-47d2-b698-618548f5162d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:40:57.545658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8fd04822-c7db-47d2-b698-618548f5162d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:57.645013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:57.680842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 838/1678] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:40:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:40:57.760789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:57.763383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:57.904583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0818aace-8200-4b94-82a7-c49de86be2c9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:57.909806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fda99cf-25a2-4b9c-ae4a-3443f6273196 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:57.910597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fda99cf-25a2-4b9c-ae4a-3443f6273196 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:57.911021 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fda99cf-25a2-4b9c-ae4a-3443f6273196 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:57.911392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fda99cf-25a2-4b9c-ae4a-3443f6273196 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:57.937893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fda99cf-25a2-4b9c-ae4a-3443f6273196 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:57.939555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fda99cf-25a2-4b9c-ae4a-3443f6273196 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:57.970829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 841/1679] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:40:58.329763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd04822-c7db-47d2-b698-618548f5162d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:58.331118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8fd04822-c7db-47d2-b698-618548f5162d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:58.354635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd04822-c7db-47d2-b698-618548f5162d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:58.390612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:58.391461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:58.391734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:58.392024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] PATH_INFO: `/v3/projects/36c72eadb72f43f9ab79dae5bc3ce054` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:58.395087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36c72eadb72f43f9ab79dae5bc3ce054)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:58.397017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:58.566721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36c72eadb72f43f9ab79dae5bc3ce054', 'request_id': 'req-29841eb4-ec5f-4555-86f1-334f09d16142'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:58.594595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36c72eadb72f43f9ab79dae5bc3ce054', 'request_id': 'req-29841eb4-ec5f-4555-86f1-334f09d16142'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:58.613385 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36c72eadb72f43f9ab79dae5bc3ce054', 'request_id': 'req-29841eb4-ec5f-4555-86f1-334f09d16142'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:58.654300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] The token cache is being invalidate because project 36c72eadb72f43f9ab79dae5bc3ce054 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:40:58.655499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:58.660289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:58.678602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 839/1680] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:40:58 2022] DELETE /identity/v3/projects/36c72eadb72f43f9ab79dae5bc3ce054 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:58.679191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fda99cf-25a2-4b9c-ae4a-3443f6273196 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:58.699741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fda99cf-25a2-4b9c-ae4a-3443f6273196 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:58.808336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:58.809800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:59.033794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:59.035581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:59.037508 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:59.037943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] PATH_INFO: `/v3/users/ae05283ab16e437f82eb00fd283d4057` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:59.040173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae05283ab16e437f82eb00fd283d4057)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:59.051743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:59.089112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29841eb4-ec5f-4555-86f1-334f09d16142 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:40:59.118251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:40:59.119373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:40:59.119837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:40:59.120306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:40:59.123513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:59.153132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:40:59.155884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:40:59.157137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:40:59.193526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 840/1681] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:40:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:40:59.223923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae05283ab16e437f82eb00fd283d4057'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:59.225063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae05283ab16e437f82eb00fd283d4057'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:59.241832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae05283ab16e437f82eb00fd283d4057'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:59.253659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae05283ab16e437f82eb00fd283d4057'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:59.300193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9102fa07-c3da-4184-93d5-9bd0218b1ad7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae05283ab16e437f82eb00fd283d4057'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:40:59.308452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 842/1682] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:40:58 2022] DELETE /identity/v3/users/ae05283ab16e437f82eb00fd283d4057 => generated 0 bytes in 629 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:40:59.728794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:40:59.730389 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:40:59.797949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97908491-7922-419e-86d9-e7e6a3e2552b service placement] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:00.046633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:00.048211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:00.050048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:00.050403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] PATH_INFO: `/v3/users/12cb4bbbe2d1439cabd3a6c3c3de9484` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:00.054579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] RBAC: Authorizing `identity:delete_user(user_id=12cb4bbbe2d1439cabd3a6c3c3de9484)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:00.056277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:00.074456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:00.177820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12cb4bbbe2d1439cabd3a6c3c3de9484'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:00.197787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12cb4bbbe2d1439cabd3a6c3c3de9484'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:00.198799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12cb4bbbe2d1439cabd3a6c3c3de9484'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:00.200281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12cb4bbbe2d1439cabd3a6c3c3de9484'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:00.221222 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12cb4bbbe2d1439cabd3a6c3c3de9484'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:00.265482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 841/1683] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:40:59 2022] DELETE /identity/v3/users/12cb4bbbe2d1439cabd3a6c3c3de9484 => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:00.270089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671e1f98-b5d7-4dd2-97f7-e7c73a298994 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:00.270554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671e1f98-b5d7-4dd2-97f7-e7c73a298994 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:00.270765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671e1f98-b5d7-4dd2-97f7-e7c73a298994 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:00.340914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-671e1f98-b5d7-4dd2-97f7-e7c73a298994 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:00.343816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-671e1f98-b5d7-4dd2-97f7-e7c73a298994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:00.540585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 843/1684] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:00.572211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:00.574538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:00.672504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671e1f98-b5d7-4dd2-97f7-e7c73a298994 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:00.677162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-671e1f98-b5d7-4dd2-97f7-e7c73a298994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:00.710281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671e1f98-b5d7-4dd2-97f7-e7c73a298994 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:00.740586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:00.742455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:00.745953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:00.747268 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] PATH_INFO: `/v3/projects/ff9bc95c0e074f66a511f524154b70bf` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:00.756837 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff9bc95c0e074f66a511f524154b70bf)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:00.776822 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:00.798882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579e9121-adf9-41f0-a4b2-38097e58caae admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:00.818799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4748752-0394-42b7-872e-bb720a0c6d57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:00.820383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4748752-0394-42b7-872e-bb720a0c6d57 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:00.824624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4748752-0394-42b7-872e-bb720a0c6d57 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:00.825308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4748752-0394-42b7-872e-bb720a0c6d57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:00.854975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4748752-0394-42b7-872e-bb720a0c6d57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:00.857710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4748752-0394-42b7-872e-bb720a0c6d57 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:00.929686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 842/1685] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:00.980005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff9bc95c0e074f66a511f524154b70bf', 'request_id': 'req-a15798aa-f0c3-476f-854f-596c56218de2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:01.002962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff9bc95c0e074f66a511f524154b70bf', 'request_id': 'req-a15798aa-f0c3-476f-854f-596c56218de2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:01.017898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff9bc95c0e074f66a511f524154b70bf', 'request_id': 'req-a15798aa-f0c3-476f-854f-596c56218de2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:01.068030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] The token cache is being invalidate because project ff9bc95c0e074f66a511f524154b70bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:01.068573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:01.071695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a15798aa-f0c3-476f-854f-596c56218de2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:01.074253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 844/1686] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:00 2022] DELETE /identity/v3/projects/ff9bc95c0e074f66a511f524154b70bf => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:01.081573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272981c5-ee5c-4518-b536-521968ea9116 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:01.082189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272981c5-ee5c-4518-b536-521968ea9116 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:01.082668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272981c5-ee5c-4518-b536-521968ea9116 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:01.137520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-272981c5-ee5c-4518-b536-521968ea9116 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:01.141925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-272981c5-ee5c-4518-b536-521968ea9116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:01.367915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 843/1687] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:01.369709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85299b-0dfb-4e77-956d-c3a77d296a6e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:01.370598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85299b-0dfb-4e77-956d-c3a77d296a6e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:01.371108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea85299b-0dfb-4e77-956d-c3a77d296a6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:01.421148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-ea85299b-0dfb-4e77-956d-c3a77d296a6e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:01.425623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ea85299b-0dfb-4e77-956d-c3a77d296a6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:01.460284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 845/1688] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:01.464927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272981c5-ee5c-4518-b536-521968ea9116 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:01.494981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272981c5-ee5c-4518-b536-521968ea9116 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:01.542633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:01.542633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:01.563703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:01.564072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:01.564834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:01.565138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:01.571094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:01.626130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 844/1689] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:01.627434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea85299b-0dfb-4e77-956d-c3a77d296a6e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:01.644239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea85299b-0dfb-4e77-956d-c3a77d296a6e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:01.650821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20032e7-3efb-4746-9afd-02ecb3c67ca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:01.651509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20032e7-3efb-4746-9afd-02ecb3c67ca4 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:01.651792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20032e7-3efb-4746-9afd-02ecb3c67ca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:01.652067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20032e7-3efb-4746-9afd-02ecb3c67ca4 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:01.652944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20032e7-3efb-4746-9afd-02ecb3c67ca4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:01.655280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20032e7-3efb-4746-9afd-02ecb3c67ca4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:01.689379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:01.734100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 846/1690] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:01 2022] POST /identity/v3/projects => generated 379 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:01.734910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:01.757577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47539c6-d4f1-4bb4-af58-1a6761baa9f3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:01.793136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:01.978459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666932c-0cb9-4569-aacc-0692f65d86bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:01.980017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666932c-0cb9-4569-aacc-0692f65d86bb admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:01.980532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666932c-0cb9-4569-aacc-0692f65d86bb admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:01.981009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666932c-0cb9-4569-aacc-0692f65d86bb admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:01.981892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f666932c-0cb9-4569-aacc-0692f65d86bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:01.985047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f666932c-0cb9-4569-aacc-0692f65d86bb admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:02.053637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20032e7-3efb-4746-9afd-02ecb3c67ca4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:02.056143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 845/1691] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:01 2022] POST /identity/v3/users => generated 290 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:02.098588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20032e7-3efb-4746-9afd-02ecb3c67ca4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:02.119279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bf161b-93b9-4953-b366-909806609e9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:02.120929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf161b-93b9-4953-b366-909806609e9c admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:02.121583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf161b-93b9-4953-b366-909806609e9c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:02.122173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf161b-93b9-4953-b366-909806609e9c admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:02.123314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bf161b-93b9-4953-b366-909806609e9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:02.139703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bf161b-93b9-4953-b366-909806609e9c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:02.188921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 847/1692] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:02 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:02.194413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666932c-0cb9-4569-aacc-0692f65d86bb admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:02.230383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666932c-0cb9-4569-aacc-0692f65d86bb admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:02.256121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:02.256969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:02.257430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:02.257961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:02.259183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:02.272628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:02.315607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 846/1693] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:02 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:02.320813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bf161b-93b9-4953-b366-909806609e9c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:02.356375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bf161b-93b9-4953-b366-909806609e9c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:02.363293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daea58b6-a689-4b86-b492-9a2e61b93d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:02.364345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daea58b6-a689-4b86-b492-9a2e61b93d51 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:02.365417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daea58b6-a689-4b86-b492-9a2e61b93d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:02.365905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daea58b6-a689-4b86-b492-9a2e61b93d51 admin admin] PATH_INFO: `/v3/projects/e2dc552429b044f7810faea6e1ebd40d/users/3b0f415db6cc4ae6a4a686a53060fa99/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:02.428643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daea58b6-a689-4b86-b492-9a2e61b93d51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2dc552429b044f7810faea6e1ebd40d, user_id=3b0f415db6cc4ae6a4a686a53060fa99, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:02.436999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daea58b6-a689-4b86-b492-9a2e61b93d51 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:02.473143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 848/1694] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:02 2022] PUT /identity/v3/projects/e2dc552429b044f7810faea6e1ebd40d/users/3b0f415db6cc4ae6a4a686a53060fa99/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:02.485462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:02.486929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33601bc-f929-4a86-be8f-299d39f95a31 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:02.487554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33601bc-f929-4a86-be8f-299d39f95a31 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:02.488356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33601bc-f929-4a86-be8f-299d39f95a31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:02.488745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:02.549452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd1698f-140e-4831-9c22-f1556c0ee63a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:02.561113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c33601bc-f929-4a86-be8f-299d39f95a31 None None] MFA Rules not processed for user `70da750fbe764462b50dc0fcef3eb0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:02.563992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c33601bc-f929-4a86-be8f-299d39f95a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:02.720853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:02.727165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:02.728375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:02.730528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] PATH_INFO: `/v3/projects/7b6cabe35cad479ba16eb542a8972313` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:02.740660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b6cabe35cad479ba16eb542a8972313)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:02.751422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 849/1695] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:02.751956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:02.758017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfaeb0b-b0ab-4719-91f0-95d5626d8215 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:02.763521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfaeb0b-b0ab-4719-91f0-95d5626d8215 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:02.763843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfaeb0b-b0ab-4719-91f0-95d5626d8215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:02.864601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-fbfaeb0b-b0ab-4719-91f0-95d5626d8215 None None] MFA Rules not processed for user `3b0f415db6cc4ae6a4a686a53060fa99`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:02.868500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fbfaeb0b-b0ab-4719-91f0-95d5626d8215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:02.926858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b6cabe35cad479ba16eb542a8972313', 'request_id': 'req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:02.954769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b6cabe35cad479ba16eb542a8972313', 'request_id': 'req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:02.970551 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b6cabe35cad479ba16eb542a8972313', 'request_id': 'req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:03.028052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] The token cache is being invalidate because project 7b6cabe35cad479ba16eb542a8972313 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:03.028911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:03.031131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:03.035971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 847/1696] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:02 2022] DELETE /identity/v3/projects/7b6cabe35cad479ba16eb542a8972313 => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:03.040321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:03.043048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:03.090114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 850/1697] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:02 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:03.091481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50bb475-ff5e-4bf6-af74-714abe1641fd None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:03.091767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50bb475-ff5e-4bf6-af74-714abe1641fd None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:03.092038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50bb475-ff5e-4bf6-af74-714abe1641fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:03.125609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:03.158417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a50bb475-ff5e-4bf6-af74-714abe1641fd None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:03.161271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a50bb475-ff5e-4bf6-af74-714abe1641fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:03.232088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:03.279395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473ac9d9-3f6c-4f4b-b84d-e7ca8bd0e06e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:03.304504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fbbfa5-3cf9-452d-b261-c2b9e514d756 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:03.306320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fbbfa5-3cf9-452d-b261-c2b9e514d756 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:03.307338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fbbfa5-3cf9-452d-b261-c2b9e514d756 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:03.308311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fbbfa5-3cf9-452d-b261-c2b9e514d756 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:03.311753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-12fbbfa5-3cf9-452d-b261-c2b9e514d756 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:03.325782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 851/1698] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:03.325782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50bb475-ff5e-4bf6-af74-714abe1641fd None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:03.327758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a50bb475-ff5e-4bf6-af74-714abe1641fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:03.363692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50bb475-ff5e-4bf6-af74-714abe1641fd None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:03.375398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea5fd7-c627-42a4-afdd-240d6830c7fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:03.376027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea5fd7-c627-42a4-afdd-240d6830c7fe admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:03.376302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea5fd7-c627-42a4-afdd-240d6830c7fe admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:03.376574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea5fd7-c627-42a4-afdd-240d6830c7fe admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:03.377220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea5fd7-c627-42a4-afdd-240d6830c7fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:03.383756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea5fd7-c627-42a4-afdd-240d6830c7fe admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:03.465361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 852/1699] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:03 2022] POST /identity/v3/projects => generated 381 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:03.468179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ba0349-a62b-4782-8776-15a0db7adc83 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:03.468597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ba0349-a62b-4782-8776-15a0db7adc83 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:03.469108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ba0349-a62b-4782-8776-15a0db7adc83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:03.522905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fbbfa5-3cf9-452d-b261-c2b9e514d756 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:03.525062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fbbfa5-3cf9-452d-b261-c2b9e514d756 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:03.557662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 848/1700] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:03 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:03.567833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8437a47-a734-4acc-a98d-5a87d91bf83c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:03.569306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8437a47-a734-4acc-a98d-5a87d91bf83c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:03.587354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8437a47-a734-4acc-a98d-5a87d91bf83c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:03.662261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-43ba0349-a62b-4782-8776-15a0db7adc83 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:03.668061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-43ba0349-a62b-4782-8776-15a0db7adc83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:03.682157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f8437a47-a734-4acc-a98d-5a87d91bf83c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:03.684446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f8437a47-a734-4acc-a98d-5a87d91bf83c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:03.836943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 849/1701] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:03.839007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8437a47-a734-4acc-a98d-5a87d91bf83c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:03.846798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 853/1702] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:41:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 381 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:03.847721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ba0349-a62b-4782-8776-15a0db7adc83 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:03.849266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-43ba0349-a62b-4782-8776-15a0db7adc83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:03.881514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8437a47-a734-4acc-a98d-5a87d91bf83c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:03.900601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ba0349-a62b-4782-8776-15a0db7adc83 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:03.916128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:03.916797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:03.917074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:03.917344 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] PATH_INFO: `/v3/projects/65c54ce7f3d24882bfb25652bad3c324` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:03.924947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6497e0f6-1e96-4060-b75e-baf55511278e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:03.925587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6497e0f6-1e96-4060-b75e-baf55511278e admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:03.925687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65c54ce7f3d24882bfb25652bad3c324)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:03.925870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6497e0f6-1e96-4060-b75e-baf55511278e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:03.926143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6497e0f6-1e96-4060-b75e-baf55511278e admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:03.926891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6497e0f6-1e96-4060-b75e-baf55511278e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:03.928900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6497e0f6-1e96-4060-b75e-baf55511278e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:03.930060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:04.000344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 850/1703] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:03 2022] POST /identity/v3/users => generated 297 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:04.002642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6497e0f6-1e96-4060-b75e-baf55511278e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:04.026297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6497e0f6-1e96-4060-b75e-baf55511278e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:04.052308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65c54ce7f3d24882bfb25652bad3c324', 'request_id': 'req-cfe353b6-9313-4abe-93e5-cd5ec3530574'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:04.064661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65c54ce7f3d24882bfb25652bad3c324', 'request_id': 'req-cfe353b6-9313-4abe-93e5-cd5ec3530574'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:04.093787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65c54ce7f3d24882bfb25652bad3c324', 'request_id': 'req-cfe353b6-9313-4abe-93e5-cd5ec3530574'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:04.120273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] The token cache is being invalidate because project 65c54ce7f3d24882bfb25652bad3c324 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:04.120571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:04.121552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:04.125411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 854/1704] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:03 2022] DELETE /identity/v3/projects/65c54ce7f3d24882bfb25652bad3c324 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:04.126362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:04.130860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:04.170101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:04.191435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:04.262625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:04.263625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:04.264096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:04.265701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:04.266559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe353b6-9313-4abe-93e5-cd5ec3530574 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:04.271212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:04.271938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:04.273613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8ac285-cf20-4704-91dd-60e44d8f19d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:04.273788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:04.274554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8ac285-cf20-4704-91dd-60e44d8f19d1 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:04.275579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8ac285-cf20-4704-91dd-60e44d8f19d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:04.276170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8ac285-cf20-4704-91dd-60e44d8f19d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:04.277731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8ac285-cf20-4704-91dd-60e44d8f19d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:04.280067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8ac285-cf20-4704-91dd-60e44d8f19d1 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:04.302202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 855/1705] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:04 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:04.303603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65393fc-845c-4785-ab61-940d0d60833b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:04.304493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65393fc-845c-4785-ab61-940d0d60833b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:04.304785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65393fc-845c-4785-ab61-940d0d60833b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:04.331290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 851/1706] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:04.331290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:04.332697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d65393fc-845c-4785-ab61-940d0d60833b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:04.333616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:04.334864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d65393fc-845c-4785-ab61-940d0d60833b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:04.358966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 856/1707] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:04.359964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65393fc-845c-4785-ab61-940d0d60833b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:04.394633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65393fc-845c-4785-ab61-940d0d60833b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:04.409237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:04.410095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:04.410206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:04.410470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] PATH_INFO: `/v3/projects/30e25f6c7ade474c91942916851c81bd/users/0eea3d03955e4f4b8f2156113cfd7863/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:04.471905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30e25f6c7ade474c91942916851c81bd, user_id=0eea3d03955e4f4b8f2156113cfd7863, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:04.481155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:04.520833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 857/1708] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:04 2022] PUT /identity/v3/projects/30e25f6c7ade474c91942916851c81bd/users/0eea3d03955e4f4b8f2156113cfd7863/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:04.522127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:04.531782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:04.679712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04095ac9-6cf8-4732-9ce4-14060aed0f5f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:04.715990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26142d-23be-47d8-8ab6-e8b57fd24525 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:04.738194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff50c4c8-e56a-4460-bdd1-07c101a57b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:04.739182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff50c4c8-e56a-4460-bdd1-07c101a57b31 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:04.739579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff50c4c8-e56a-4460-bdd1-07c101a57b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:04.740043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff50c4c8-e56a-4460-bdd1-07c101a57b31 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:04.740788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff50c4c8-e56a-4460-bdd1-07c101a57b31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:04.742911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff50c4c8-e56a-4460-bdd1-07c101a57b31 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:04.833459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 858/1709] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:04 2022] POST /identity/v3/projects => generated 377 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:04.834986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46893df1-b670-47b7-8888-fd3e7de6683f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:04.835376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46893df1-b670-47b7-8888-fd3e7de6683f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:04.835775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46893df1-b670-47b7-8888-fd3e7de6683f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:04.862852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-46893df1-b670-47b7-8888-fd3e7de6683f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:04.864924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-46893df1-b670-47b7-8888-fd3e7de6683f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:04.887088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:04.888312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:04.888312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:04.888559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:04.893663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:04.955433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:04.986615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 859/1710] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:04.988426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46893df1-b670-47b7-8888-fd3e7de6683f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:05.001524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46893df1-b670-47b7-8888-fd3e7de6683f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:05.018466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3ba78a-ec3c-4b85-8347-07b91acbc09d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:05.019484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3ba78a-ec3c-4b85-8347-07b91acbc09d admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.019775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3ba78a-ec3c-4b85-8347-07b91acbc09d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.021219 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3ba78a-ec3c-4b85-8347-07b91acbc09d admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:05.022627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3ba78a-ec3c-4b85-8347-07b91acbc09d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:05.026941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3ba78a-ec3c-4b85-8347-07b91acbc09d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:05.070010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 860/1711] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:04 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:05.071940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ad521-b626-494e-a766-08d11db30421 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.072290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ad521-b626-494e-a766-08d11db30421 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.072575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ad521-b626-494e-a766-08d11db30421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:05.087939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:05.089917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:05.113117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:05.128196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 852/1712] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:41:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 801 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:41:05.130625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:05.147961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a543db21-eb12-4a1e-8ec2-bb125f1ac307 service swift] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:05.154376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab44d1d-2170-4f11-bffb-61bbd56783c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:05.156102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab44d1d-2170-4f11-bffb-61bbd56783c7 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.156593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab44d1d-2170-4f11-bffb-61bbd56783c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.159817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab44d1d-2170-4f11-bffb-61bbd56783c7 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:05.164244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab44d1d-2170-4f11-bffb-61bbd56783c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:05.170054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab44d1d-2170-4f11-bffb-61bbd56783c7 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:05.179748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-265ad521-b626-494e-a766-08d11db30421 None None] MFA Rules not processed for user `cc7aa14777ab43a083b3a120c4349d67`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:05.181952 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-265ad521-b626-494e-a766-08d11db30421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:05.214851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 853/1713] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:05 2022] POST /identity/v3/users => generated 289 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:05.217247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab44d1d-2170-4f11-bffb-61bbd56783c7 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:05.263362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab44d1d-2170-4f11-bffb-61bbd56783c7 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:05.298565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:05.304409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.304879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.306488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:05.309238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:05.309874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:05.324220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:05.376937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 854/1714] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:05.377902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:05.410296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6212a922-f89e-42cd-bc84-32cec726680d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:05.444004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8451fa93-31b6-478c-8342-361b4bdb68f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:05.445506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8451fa93-31b6-478c-8342-361b4bdb68f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.445639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8451fa93-31b6-478c-8342-361b4bdb68f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.446348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8451fa93-31b6-478c-8342-361b4bdb68f1 admin admin] PATH_INFO: `/v3/projects/30e25f6c7ade474c91942916851c81bd/users/0eea3d03955e4f4b8f2156113cfd7863/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:05.453723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8451fa93-31b6-478c-8342-361b4bdb68f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30e25f6c7ade474c91942916851c81bd, user_id=0eea3d03955e4f4b8f2156113cfd7863, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:05.456436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8451fa93-31b6-478c-8342-361b4bdb68f1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:05.480678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 861/1715] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 410 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:05.487737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160872b3-d5bf-441b-aefc-9f00b48c4ba2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.498021 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160872b3-d5bf-441b-aefc-9f00b48c4ba2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.500856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160872b3-d5bf-441b-aefc-9f00b48c4ba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:05.508223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 855/1716] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:05 2022] PUT /identity/v3/projects/30e25f6c7ade474c91942916851c81bd/users/0eea3d03955e4f4b8f2156113cfd7863/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:05.511866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8451fa93-31b6-478c-8342-361b4bdb68f1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:05.593685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-160872b3-d5bf-441b-aefc-9f00b48c4ba2 None None] MFA Rules not processed for user `003dd6612cda48618966db8c0788b794`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:05.596997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-160872b3-d5bf-441b-aefc-9f00b48c4ba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:05.728467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8451fa93-31b6-478c-8342-361b4bdb68f1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:05.745013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f8befe-c20e-46c5-bf6f-1ab4cfe8c167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:05.748911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f8befe-c20e-46c5-bf6f-1ab4cfe8c167 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.751300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f8befe-c20e-46c5-bf6f-1ab4cfe8c167 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.752503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f8befe-c20e-46c5-bf6f-1ab4cfe8c167 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:05.757623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f8befe-c20e-46c5-bf6f-1ab4cfe8c167 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:05.761373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f8befe-c20e-46c5-bf6f-1ab4cfe8c167 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:05.800742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 856/1717] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:05 2022] GET /identity/v3/roles => generated 1420 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:05.812418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f8befe-c20e-46c5-bf6f-1ab4cfe8c167 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:05.846050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f8befe-c20e-46c5-bf6f-1ab4cfe8c167 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:05.861011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dcd8cb-e11e-4270-9509-6d0ba5596a16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:05.863194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dcd8cb-e11e-4270-9509-6d0ba5596a16 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.866563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dcd8cb-e11e-4270-9509-6d0ba5596a16 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.867069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dcd8cb-e11e-4270-9509-6d0ba5596a16 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:05.868416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dcd8cb-e11e-4270-9509-6d0ba5596a16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:05.878857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dcd8cb-e11e-4270-9509-6d0ba5596a16 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:05.894824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 862/1718] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:05 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 413 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:05.901897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160872b3-d5bf-441b-aefc-9f00b48c4ba2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:05.907601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-160872b3-d5bf-441b-aefc-9f00b48c4ba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:05.985567 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 857/1719] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:05 2022] POST /identity/v3/projects => generated 403 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:05.987506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f9f21f-fb0d-42f3-9200-44fe366253a0 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:05.987932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f9f21f-fb0d-42f3-9200-44fe366253a0 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:05.988228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f9f21f-fb0d-42f3-9200-44fe366253a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:06.125311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c8f9f21f-fb0d-42f3-9200-44fe366253a0 None None] MFA Rules not processed for user `0eea3d03955e4f4b8f2156113cfd7863`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:06.129419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c8f9f21f-fb0d-42f3-9200-44fe366253a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:06.196391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160872b3-d5bf-441b-aefc-9f00b48c4ba2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:06.205827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:06.440164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 858/1720] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:05 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 454 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:06.441092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f9f21f-fb0d-42f3-9200-44fe366253a0 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:06.452613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b5ac6b-e743-457c-ab51-18cafa4ff8e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:06.452613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5ac6b-e743-457c-ab51-18cafa4ff8e5 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:06.452613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5ac6b-e743-457c-ab51-18cafa4ff8e5 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:06.452613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b5ac6b-e743-457c-ab51-18cafa4ff8e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:06.523495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:06.688676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b5ac6b-e743-457c-ab51-18cafa4ff8e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:06.702858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b5ac6b-e743-457c-ab51-18cafa4ff8e5 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:06.732522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f9f21f-fb0d-42f3-9200-44fe366253a0 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:06.738400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 863/1721] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:05 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 843 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:06.740004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b5ac6b-e743-457c-ab51-18cafa4ff8e5 service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:06.745487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:06.746492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:06.750444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:06.750922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:06.751774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:06.754447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:06.771089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b5ac6b-e743-457c-ab51-18cafa4ff8e5 service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:06.781909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 859/1722] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:06 2022] GET /identity/v3/roles => generated 1420 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:06.783732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:06.785177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:06.786105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:06.786530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:06.787583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:06.788396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:06.798153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:06.806437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:06.913944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 864/1723] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:06 2022] POST /identity/v3/users => generated 302 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:06.918006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:06.922719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:07.074198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3d3167-b969-4aff-b7bf-e77a16fe6c87 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:07.107683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8dc511-bf49-4ac8-a619-513520f173db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:07.108836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8dc511-bf49-4ac8-a619-513520f173db service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:07.124294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8dc511-bf49-4ac8-a619-513520f173db service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:07.124294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8dc511-bf49-4ac8-a619-513520f173db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:07.182750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7a05f2-6f84-4986-a990-1e7408c9248c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:07.207689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b66fa9-d3f8-48b2-a5a5-51f53f51feaf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:07.211026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b66fa9-d3f8-48b2-a5a5-51f53f51feaf service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:07.212523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b66fa9-d3f8-48b2-a5a5-51f53f51feaf service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:07.214908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b66fa9-d3f8-48b2-a5a5-51f53f51feaf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:07.271275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b66fa9-d3f8-48b2-a5a5-51f53f51feaf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:07.280728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b66fa9-d3f8-48b2-a5a5-51f53f51feaf service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:07.353235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 860/1724] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:06 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 569 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:07.355136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b66fa9-d3f8-48b2-a5a5-51f53f51feaf service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:07.409237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8dc511-bf49-4ac8-a619-513520f173db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:07.413549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8dc511-bf49-4ac8-a619-513520f173db service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:07.425541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b66fa9-d3f8-48b2-a5a5-51f53f51feaf service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:07.453755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a769f37-5923-45d0-a3cf-3863261b0b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:07.456403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a769f37-5923-45d0-a3cf-3863261b0b2e admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:07.458705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a769f37-5923-45d0-a3cf-3863261b0b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:07.458705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a769f37-5923-45d0-a3cf-3863261b0b2e admin admin] PATH_INFO: `/v3/projects/5c029a4bb1ce4d48abae261c46a905f5/users/7e9057f1e5704e44b8951d26295d2ca3/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:07.507320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 865/1725] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:06 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 593 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:07.520686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8dc511-bf49-4ac8-a619-513520f173db service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:07.564949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a769f37-5923-45d0-a3cf-3863261b0b2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c029a4bb1ce4d48abae261c46a905f5, user_id=7e9057f1e5704e44b8951d26295d2ca3, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:07.586006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a769f37-5923-45d0-a3cf-3863261b0b2e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:07.613891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8dc511-bf49-4ac8-a619-513520f173db service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:07.652865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 861/1726] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:07 2022] PUT /identity/v3/projects/5c029a4bb1ce4d48abae261c46a905f5/users/7e9057f1e5704e44b8951d26295d2ca3/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:07.664838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fceb880-aa6b-4acc-9e11-c6c61edc8730 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:07.666856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fceb880-aa6b-4acc-9e11-c6c61edc8730 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:07.667556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fceb880-aa6b-4acc-9e11-c6c61edc8730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:07.680028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac352c2d-3abf-402f-b10a-392a92afb16d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:07.680674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac352c2d-3abf-402f-b10a-392a92afb16d admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:07.680961 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac352c2d-3abf-402f-b10a-392a92afb16d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:07.681238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac352c2d-3abf-402f-b10a-392a92afb16d admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:07.697314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac352c2d-3abf-402f-b10a-392a92afb16d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:07.705207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac352c2d-3abf-402f-b10a-392a92afb16d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:07.769746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 866/1727] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:07 2022] GET /identity/v3/roles => generated 1420 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:07.771622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d567e9-31bd-485a-ab0c-dd761520503f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:07.772567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d567e9-31bd-485a-ab0c-dd761520503f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:07.776339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d567e9-31bd-485a-ab0c-dd761520503f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:07.798758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-6fceb880-aa6b-4acc-9e11-c6c61edc8730 None None] MFA Rules not processed for user `dea13198ef8e45de85b92e79e3ee71b0`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:07.806446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6fceb880-aa6b-4acc-9e11-c6c61edc8730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:07.931860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:07.944782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-79d567e9-31bd-485a-ab0c-dd761520503f None None] MFA Rules not processed for user `7e9057f1e5704e44b8951d26295d2ca3`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:07.950587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-79d567e9-31bd-485a-ab0c-dd761520503f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:07.995026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 862/1728] 10.222.0.84 () {58 vars in 1015 bytes} [Mon Aug 29 22:41:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 336 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:07.996058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fceb880-aa6b-4acc-9e11-c6c61edc8730 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:08.154491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:08.200074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fceb880-aa6b-4acc-9e11-c6c61edc8730 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:08.214272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:08.214955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:08.219454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:08.219741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:08.225614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:08.284726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 867/1729] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 514 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:08.297722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d567e9-31bd-485a-ab0c-dd761520503f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:08.552828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:08.554662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:08.588107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 863/1730] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:07 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 592 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:08.588880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:08.595324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:08.624947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f932c46-83cf-4a1b-9255-62c277cc75b7 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:08.636891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d567e9-31bd-485a-ab0c-dd761520503f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:08.639915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906050e6-cfbf-4d59-941e-642917417967 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:08.643605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906050e6-cfbf-4d59-941e-642917417967 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:08.647308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906050e6-cfbf-4d59-941e-642917417967 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:08.647308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906050e6-cfbf-4d59-941e-642917417967 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:08.661600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6f941e-c8f2-426f-90e7-0bc644d6b395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:08.662243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6f941e-c8f2-426f-90e7-0bc644d6b395 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:08.662527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6f941e-c8f2-426f-90e7-0bc644d6b395 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:08.662844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6f941e-c8f2-426f-90e7-0bc644d6b395 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:08.663512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6f941e-c8f2-426f-90e7-0bc644d6b395 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:08.663950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906050e6-cfbf-4d59-941e-642917417967 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:08.665144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6f941e-c8f2-426f-90e7-0bc644d6b395 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:08.668729 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906050e6-cfbf-4d59-941e-642917417967 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:08.705671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 864/1731] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:08.707194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 868/1732] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:08 2022] GET /identity/v3/roles => generated 1420 bytes in 422 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:08.712483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906050e6-cfbf-4d59-941e-642917417967 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:08.712735 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6f941e-c8f2-426f-90e7-0bc644d6b395 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:08.751830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906050e6-cfbf-4d59-941e-642917417967 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:08.759777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6f941e-c8f2-426f-90e7-0bc644d6b395 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:08.773487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eca9e5-5ba7-45eb-8297-f0ced2d8ebeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:08.774852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eca9e5-5ba7-45eb-8297-f0ced2d8ebeb admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:08.776836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eca9e5-5ba7-45eb-8297-f0ced2d8ebeb admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:08.778373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eca9e5-5ba7-45eb-8297-f0ced2d8ebeb admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:08.780017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eca9e5-5ba7-45eb-8297-f0ced2d8ebeb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:08.781632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae38d92-09ad-4b8a-85f1-1af3d07f4cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:08.790323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eca9e5-5ba7-45eb-8297-f0ced2d8ebeb admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:08.791186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae38d92-09ad-4b8a-85f1-1af3d07f4cc1 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:08.791902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae38d92-09ad-4b8a-85f1-1af3d07f4cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:08.792936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae38d92-09ad-4b8a-85f1-1af3d07f4cc1 admin admin] PATH_INFO: `/v3/projects/a7e1d9b8ca5f4f5a95684a15e7413d9e/users/aef7c3637dae4a70864e1d49804bd783/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:08.882974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae38d92-09ad-4b8a-85f1-1af3d07f4cc1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7e1d9b8ca5f4f5a95684a15e7413d9e, user_id=aef7c3637dae4a70864e1d49804bd783, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:08.904411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae38d92-09ad-4b8a-85f1-1af3d07f4cc1 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:08.913107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 865/1733] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:08 2022] POST /identity/v3/projects => generated 377 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:08.914432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eca9e5-5ba7-45eb-8297-f0ced2d8ebeb admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:08.975313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 869/1734] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:08 2022] PUT /identity/v3/projects/a7e1d9b8ca5f4f5a95684a15e7413d9e/users/aef7c3637dae4a70864e1d49804bd783/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:08.981701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee25925-682e-43fc-aa5f-e9b0e4f61e06 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:08.982042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee25925-682e-43fc-aa5f-e9b0e4f61e06 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:08.982246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee25925-682e-43fc-aa5f-e9b0e4f61e06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:09.068081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9ee25925-682e-43fc-aa5f-e9b0e4f61e06 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:09.071988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9ee25925-682e-43fc-aa5f-e9b0e4f61e06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:09.195993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eca9e5-5ba7-45eb-8297-f0ced2d8ebeb admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:09.319690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 870/1735] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:09.384624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee25925-682e-43fc-aa5f-e9b0e4f61e06 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:09.386777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9ee25925-682e-43fc-aa5f-e9b0e4f61e06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:09.470356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:09.528250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db2a5d7-1aff-4d30-b87e-757c1ebc35c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:09.529521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db2a5d7-1aff-4d30-b87e-757c1ebc35c2 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:09.530008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db2a5d7-1aff-4d30-b87e-757c1ebc35c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:09.530984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db2a5d7-1aff-4d30-b87e-757c1ebc35c2 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:09.532488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db2a5d7-1aff-4d30-b87e-757c1ebc35c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:09.542989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db2a5d7-1aff-4d30-b87e-757c1ebc35c2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:09.588703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:09.634297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 866/1736] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:08 2022] POST /identity/v3/users => generated 295 bytes in 720 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:09.639776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db2a5d7-1aff-4d30-b87e-757c1ebc35c2 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:09.652387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee25925-682e-43fc-aa5f-e9b0e4f61e06 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:09.679169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db2a5d7-1aff-4d30-b87e-757c1ebc35c2 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:09.681233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd2e55-af87-4b64-881e-7576f2ac7aeb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:09.681982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd2e55-af87-4b64-881e-7576f2ac7aeb service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:09.682379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd2e55-af87-4b64-881e-7576f2ac7aeb service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:09.683565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd2e55-af87-4b64-881e-7576f2ac7aeb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:09.690669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95abda32-9ddf-42e3-af49-31b935a0c95d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:09.691502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95abda32-9ddf-42e3-af49-31b935a0c95d admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:09.691952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95abda32-9ddf-42e3-af49-31b935a0c95d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:09.692395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95abda32-9ddf-42e3-af49-31b935a0c95d admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:09.693211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95abda32-9ddf-42e3-af49-31b935a0c95d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:09.696864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95abda32-9ddf-42e3-af49-31b935a0c95d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:09.725523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fd2e55-af87-4b64-881e-7576f2ac7aeb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:09.731392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fd2e55-af87-4b64-881e-7576f2ac7aeb service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:09.737548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 867/1737] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:09 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:09.741439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95abda32-9ddf-42e3-af49-31b935a0c95d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:09.779299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 871/1738] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:09.781529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95abda32-9ddf-42e3-af49-31b935a0c95d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:09.791011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef48a7c-4a57-49d5-b8dc-9998407f4fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:09.792074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef48a7c-4a57-49d5-b8dc-9998407f4fb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:09.792533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef48a7c-4a57-49d5-b8dc-9998407f4fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:09.792999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef48a7c-4a57-49d5-b8dc-9998407f4fb2 admin admin] PATH_INFO: `/v3/projects/0f8147eaf48b4aa0af74d4925ef26a66/users/66ad8a4b41104aab8bdc818bbff338ca/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:09.836160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef48a7c-4a57-49d5-b8dc-9998407f4fb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f8147eaf48b4aa0af74d4925ef26a66, user_id=66ad8a4b41104aab8bdc818bbff338ca, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:09.840116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef48a7c-4a57-49d5-b8dc-9998407f4fb2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:09.884854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 868/1739] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:09 2022] PUT /identity/v3/projects/0f8147eaf48b4aa0af74d4925ef26a66/users/66ad8a4b41104aab8bdc818bbff338ca/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:09.889414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd2e55-af87-4b64-881e-7576f2ac7aeb service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:09.901293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd2e55-af87-4b64-881e-7576f2ac7aeb service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:10.204727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84e2081-9a90-47fa-9a52-0cd93a66292e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:10.205294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e2081-9a90-47fa-9a52-0cd93a66292e admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:10.205743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e2081-9a90-47fa-9a52-0cd93a66292e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:10.206178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84e2081-9a90-47fa-9a52-0cd93a66292e admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:10.207218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84e2081-9a90-47fa-9a52-0cd93a66292e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:10.209037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84e2081-9a90-47fa-9a52-0cd93a66292e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:10.259038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef48a7c-4a57-49d5-b8dc-9998407f4fb2 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:10.277704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 872/1740] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:09 2022] GET /identity/v3/roles => generated 1420 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:10.281710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef48a7c-4a57-49d5-b8dc-9998407f4fb2 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:10.299678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:10.300373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:10.300654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:10.300926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] PATH_INFO: `/v3/projects/0f8147eaf48b4aa0af74d4925ef26a66/users/66ad8a4b41104aab8bdc818bbff338ca/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:10.307573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f8147eaf48b4aa0af74d4925ef26a66, user_id=66ad8a4b41104aab8bdc818bbff338ca, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:10.312853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:10.368635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 869/1741] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:10 2022] PUT /identity/v3/projects/0f8147eaf48b4aa0af74d4925ef26a66/users/66ad8a4b41104aab8bdc818bbff338ca/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:10.389822 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06d9b2-7214-424c-af94-afa813ee880e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:10.390383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06d9b2-7214-424c-af94-afa813ee880e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:10.390881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb06d9b2-7214-424c-af94-afa813ee880e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:10.591723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-fb06d9b2-7214-424c-af94-afa813ee880e None None] MFA Rules not processed for user `66ad8a4b41104aab8bdc818bbff338ca`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:10.593835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fb06d9b2-7214-424c-af94-afa813ee880e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:11.123527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 873/1742] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:10 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 735 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:11.313922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:11.317526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:11.617453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b7a55c-f1c9-4eb3-98f6-5fdaa9e5fd9f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:11.636216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5760bd33-aa5c-4137-9e5d-12c86c3dd59c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:11.637020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5760bd33-aa5c-4137-9e5d-12c86c3dd59c service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:11.637537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5760bd33-aa5c-4137-9e5d-12c86c3dd59c service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:11.638032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5760bd33-aa5c-4137-9e5d-12c86c3dd59c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:11.868747 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5760bd33-aa5c-4137-9e5d-12c86c3dd59c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:11.870981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5760bd33-aa5c-4137-9e5d-12c86c3dd59c service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:11.897999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 870/1743] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:11 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 584 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:14.458590 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7659c954-993a-45c1-b7cf-90b123898d94 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:14.460318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7659c954-993a-45c1-b7cf-90b123898d94 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:14.461056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7659c954-993a-45c1-b7cf-90b123898d94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:14.522770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7659c954-993a-45c1-b7cf-90b123898d94 None None] MFA Rules not processed for user `70da750fbe764462b50dc0fcef3eb0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:14.527937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7659c954-993a-45c1-b7cf-90b123898d94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:14.697307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:14.819018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 874/1744] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 361 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:14.828987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f08f50-dd98-4e96-9d40-a967a715d814 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:14.829876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f08f50-dd98-4e96-9d40-a967a715d814 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:14.830366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f08f50-dd98-4e96-9d40-a967a715d814 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:14.832383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 871/1745] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:41:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:41:14.839566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c119758-6282-42bd-97a6-8700874fba24 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:14.839887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c119758-6282-42bd-97a6-8700874fba24 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:14.840122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c119758-6282-42bd-97a6-8700874fba24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:14.873947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7c119758-6282-42bd-97a6-8700874fba24 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:14.875231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7c119758-6282-42bd-97a6-8700874fba24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:14.941587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc49276-b994-412d-bb73-7a021d380ed4 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:14.941871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc49276-b994-412d-bb73-7a021d380ed4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:14.942148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc49276-b994-412d-bb73-7a021d380ed4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:14.980339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-0dc49276-b994-412d-bb73-7a021d380ed4 None None] MFA Rules not processed for user `003dd6612cda48618966db8c0788b794`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:14.993613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0dc49276-b994-412d-bb73-7a021d380ed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:15.132568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 872/1746] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:14 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:15.134181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5720503a-b28e-409a-b45c-a995420be705 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:15.134646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5720503a-b28e-409a-b45c-a995420be705 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:15.135111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5720503a-b28e-409a-b45c-a995420be705 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:15.140744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 875/1747] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:41:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:15.161677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:15.219029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5720503a-b28e-409a-b45c-a995420be705 None None] MFA Rules not processed for user `aef7c3637dae4a70864e1d49804bd783`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:15.220821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5720503a-b28e-409a-b45c-a995420be705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:15.441158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 873/1748] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:15 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:15.446067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c119758-6282-42bd-97a6-8700874fba24 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:15.600176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c119758-6282-42bd-97a6-8700874fba24 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:15.607420 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4390989c-2430-4cd9-a13f-f86578e7e9b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:15.608871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390989c-2430-4cd9-a13f-f86578e7e9b4 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:15.609186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390989c-2430-4cd9-a13f-f86578e7e9b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:15.609542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390989c-2430-4cd9-a13f-f86578e7e9b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:15.610355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4390989c-2430-4cd9-a13f-f86578e7e9b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:15.612556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4390989c-2430-4cd9-a13f-f86578e7e9b4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:15.681755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5720503a-b28e-409a-b45c-a995420be705 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:15.694572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 876/1749] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:15 2022] POST /identity/v3/projects => generated 405 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:15.699779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5720503a-b28e-409a-b45c-a995420be705 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:15.723228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799d519-1ce6-40c7-9bcf-acefbb044210 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:15.723518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799d519-1ce6-40c7-9bcf-acefbb044210 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:15.723800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799d519-1ce6-40c7-9bcf-acefbb044210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:15.781605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1799d519-1ce6-40c7-9bcf-acefbb044210 None None] MFA Rules not processed for user `cc7aa14777ab43a083b3a120c4349d67`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:15.784329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1799d519-1ce6-40c7-9bcf-acefbb044210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:15.865125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b1bfcb-9c47-429d-8923-e21c3062c40b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:15.866255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b1bfcb-9c47-429d-8923-e21c3062c40b admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:15.866547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b1bfcb-9c47-429d-8923-e21c3062c40b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:15.866880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b1bfcb-9c47-429d-8923-e21c3062c40b admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:15.867842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b1bfcb-9c47-429d-8923-e21c3062c40b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:15.871299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b1bfcb-9c47-429d-8923-e21c3062c40b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:15.917782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 874/1750] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:15 2022] POST /identity/v3/users => generated 309 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:15.919447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b1bfcb-9c47-429d-8923-e21c3062c40b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:15.921560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 877/1751] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:15 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:15.938127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b1bfcb-9c47-429d-8923-e21c3062c40b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:15.947717 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf252bde-3b28-401c-a104-caa320ae9d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:15.947967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf252bde-3b28-401c-a104-caa320ae9d5e admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:15.948711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf252bde-3b28-401c-a104-caa320ae9d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:15.948711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf252bde-3b28-401c-a104-caa320ae9d5e admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:15.949396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf252bde-3b28-401c-a104-caa320ae9d5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:15.951110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf252bde-3b28-401c-a104-caa320ae9d5e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:15.962301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 875/1752] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:15 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:15.968510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799d519-1ce6-40c7-9bcf-acefbb044210 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:15.990350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799d519-1ce6-40c7-9bcf-acefbb044210 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:15.995812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45110145-5183-4051-a528-cd31e972c3d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:15.996947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45110145-5183-4051-a528-cd31e972c3d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:15.997412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45110145-5183-4051-a528-cd31e972c3d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:15.997866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45110145-5183-4051-a528-cd31e972c3d6 admin admin] PATH_INFO: `/v3/projects/fe0bf0dc097b43678c10294dfe4a39dd/users/35e573bb1e534703a44ac74c245a18f1/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:16.006668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:16.025818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45110145-5183-4051-a528-cd31e972c3d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe0bf0dc097b43678c10294dfe4a39dd, user_id=35e573bb1e534703a44ac74c245a18f1, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:16.030303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45110145-5183-4051-a528-cd31e972c3d6 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:16.054828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 878/1753] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:15 2022] PUT /identity/v3/projects/fe0bf0dc097b43678c10294dfe4a39dd/users/35e573bb1e534703a44ac74c245a18f1/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:16.057011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf252bde-3b28-401c-a104-caa320ae9d5e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:16.071894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf252bde-3b28-401c-a104-caa320ae9d5e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:16.185611 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d714c54-b08a-47d4-86ac-d5a7ed85b7d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:16.186636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d714c54-b08a-47d4-86ac-d5a7ed85b7d4 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:16.187185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d714c54-b08a-47d4-86ac-d5a7ed85b7d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:16.187698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d714c54-b08a-47d4-86ac-d5a7ed85b7d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:16.189124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d714c54-b08a-47d4-86ac-d5a7ed85b7d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:16.191278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d714c54-b08a-47d4-86ac-d5a7ed85b7d4 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:16.211881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 876/1754] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:16 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:16.212728 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45110145-5183-4051-a528-cd31e972c3d6 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:16.227933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45110145-5183-4051-a528-cd31e972c3d6 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:16.234618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0171df63-9a77-428a-a5f7-a6cce4b3048e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:16.235956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0171df63-9a77-428a-a5f7-a6cce4b3048e admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:16.236551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0171df63-9a77-428a-a5f7-a6cce4b3048e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:16.237231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0171df63-9a77-428a-a5f7-a6cce4b3048e admin admin] PATH_INFO: `/v3/projects/fe0bf0dc097b43678c10294dfe4a39dd/users/35e573bb1e534703a44ac74c245a18f1/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:16.239605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0171df63-9a77-428a-a5f7-a6cce4b3048e admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe0bf0dc097b43678c10294dfe4a39dd, user_id=35e573bb1e534703a44ac74c245a18f1, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:16.241970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2e14c1-059f-4fa9-a5ee-7d6e6e67aeba None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:16.242269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2e14c1-059f-4fa9-a5ee-7d6e6e67aeba None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:16.242579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2e14c1-059f-4fa9-a5ee-7d6e6e67aeba None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:16.243418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0171df63-9a77-428a-a5f7-a6cce4b3048e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:16.246139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 877/1755] 10.222.0.77 () {54 vars in 912 bytes} [Mon Aug 29 22:41:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:41:16.254101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ee6e14-a7d2-4be4-9409-0ef09278fd92 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:16.254430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ee6e14-a7d2-4be4-9409-0ef09278fd92 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:16.254685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ee6e14-a7d2-4be4-9409-0ef09278fd92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:16.261157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 879/1756] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:16 2022] PUT /identity/v3/projects/fe0bf0dc097b43678c10294dfe4a39dd/users/35e573bb1e534703a44ac74c245a18f1/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:16.283936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-10ee6e14-a7d2-4be4-9409-0ef09278fd92 None None] MFA Rules not processed for user `dea13198ef8e45de85b92e79e3ee71b0`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:16.285512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-10ee6e14-a7d2-4be4-9409-0ef09278fd92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:16.292737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:16.386693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 878/1757] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:41:16 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:16.408516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0171df63-9a77-428a-a5f7-a6cce4b3048e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:16.427975 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0171df63-9a77-428a-a5f7-a6cce4b3048e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:16.431945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ee6e14-a7d2-4be4-9409-0ef09278fd92 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:16.448015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006c7233-9fa6-4a29-8834-b15e3cab0d14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:16.448656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c7233-9fa6-4a29-8834-b15e3cab0d14 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:16.448936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c7233-9fa6-4a29-8834-b15e3cab0d14 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:16.449213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c7233-9fa6-4a29-8834-b15e3cab0d14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:16.496725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006c7233-9fa6-4a29-8834-b15e3cab0d14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:16.505682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006c7233-9fa6-4a29-8834-b15e3cab0d14 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:16.527345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 880/1758] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:16 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:16.750070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ee6e14-a7d2-4be4-9409-0ef09278fd92 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:16.771672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a55ac3-261f-4112-accb-d2d8f66e9210 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:16.772652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a55ac3-261f-4112-accb-d2d8f66e9210 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:16.773163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a55ac3-261f-4112-accb-d2d8f66e9210 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:16.773774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a55ac3-261f-4112-accb-d2d8f66e9210 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:16.825114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a55ac3-261f-4112-accb-d2d8f66e9210 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:16.827587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a55ac3-261f-4112-accb-d2d8f66e9210 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:16.880086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 879/1759] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:41:18.771439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efedbccf-e378-4fe8-a8c6-34d704f35a41 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:18.771972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efedbccf-e378-4fe8-a8c6-34d704f35a41 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:18.771972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efedbccf-e378-4fe8-a8c6-34d704f35a41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:18.803253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-efedbccf-e378-4fe8-a8c6-34d704f35a41 None None] MFA Rules not processed for user `35e573bb1e534703a44ac74c245a18f1`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:18.804618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-efedbccf-e378-4fe8-a8c6-34d704f35a41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:18.898082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 881/1760] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:18 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:19.895454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a4bef1-52ee-42dc-a79c-336f03cb0529 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:19.896058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a4bef1-52ee-42dc-a79c-336f03cb0529 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:19.896058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a4bef1-52ee-42dc-a79c-336f03cb0529 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:19.897374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 880/1761] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:41:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:41:19.912852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7aac2c-7ac3-4a47-a379-b802194a9197 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:19.914110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7aac2c-7ac3-4a47-a379-b802194a9197 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:19.914612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7aac2c-7ac3-4a47-a379-b802194a9197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:19.943683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-ff7aac2c-7ac3-4a47-a379-b802194a9197 None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:19.945012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ff7aac2c-7ac3-4a47-a379-b802194a9197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:20.069437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a4bef1-52ee-42dc-a79c-336f03cb0529 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:20.143360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 882/1762] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:41:19 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:20.157734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb4afb-06ce-464f-86d2-72cc796838e6 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:20.158034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb4afb-06ce-464f-86d2-72cc796838e6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:20.160560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb4afb-06ce-464f-86d2-72cc796838e6 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:20.166285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 883/1763] 10.222.0.77 () {54 vars in 961 bytes} [Mon Aug 29 22:41:20 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:41:20.168005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a4bef1-52ee-42dc-a79c-336f03cb0529 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:20.172966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316fcee0-6b38-4003-9b56-0cf7caefc669 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:20.173141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882bafde-b80e-458b-af75-55557422ba9b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:20.173442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882bafde-b80e-458b-af75-55557422ba9b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:20.173729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882bafde-b80e-458b-af75-55557422ba9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:20.174411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316fcee0-6b38-4003-9b56-0cf7caefc669 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:20.174902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316fcee0-6b38-4003-9b56-0cf7caefc669 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:20.175321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316fcee0-6b38-4003-9b56-0cf7caefc669 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:20.251365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-882bafde-b80e-458b-af75-55557422ba9b None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:20.252640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-882bafde-b80e-458b-af75-55557422ba9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:20.287225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 884/1764] 10.222.0.77 () {58 vars in 1058 bytes} [Mon Aug 29 22:41:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:20.292877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:20.297510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316fcee0-6b38-4003-9b56-0cf7caefc669 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:20.298292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882bafde-b80e-458b-af75-55557422ba9b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:20.299003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316fcee0-6b38-4003-9b56-0cf7caefc669 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:20.317871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882bafde-b80e-458b-af75-55557422ba9b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:20.321029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 881/1765] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:20 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:20.330095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3085c4-005c-41cb-b2d3-b3e772bca3b7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:20.332469 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3085c4-005c-41cb-b2d3-b3e772bca3b7 service placement] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:20.333613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3085c4-005c-41cb-b2d3-b3e772bca3b7 service placement] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:20.334993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3085c4-005c-41cb-b2d3-b3e772bca3b7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:20.361763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3085c4-005c-41cb-b2d3-b3e772bca3b7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:20.363371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3085c4-005c-41cb-b2d3-b3e772bca3b7 service placement] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:20.439868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 885/1766] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:21.245203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316fcee0-6b38-4003-9b56-0cf7caefc669 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:21.335956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316fcee0-6b38-4003-9b56-0cf7caefc669 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:21.342693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:21.343553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:21.343835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:21.344282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] PATH_INFO: `/v3/users/aef7c3637dae4a70864e1d49804bd783` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:21.345756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aef7c3637dae4a70864e1d49804bd783)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:21.347757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:21.407062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aef7c3637dae4a70864e1d49804bd783'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:21.413245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aef7c3637dae4a70864e1d49804bd783'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:21.414078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aef7c3637dae4a70864e1d49804bd783'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:21.414772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aef7c3637dae4a70864e1d49804bd783'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:21.424460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aef7c3637dae4a70864e1d49804bd783'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:21.446817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 882/1767] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:21 2022] DELETE /identity/v3/users/aef7c3637dae4a70864e1d49804bd783 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:21.850360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3085c4-005c-41cb-b2d3-b3e772bca3b7 service placement] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:21.865566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3085c4-005c-41cb-b2d3-b3e772bca3b7 service placement] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:21.897852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:21.970382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:21.971057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:21.971357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:21.971615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] PATH_INFO: `/v3/projects/a7e1d9b8ca5f4f5a95684a15e7413d9e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:21.972795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7e1d9b8ca5f4f5a95684a15e7413d9e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:21.975074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:22.043011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7e1d9b8ca5f4f5a95684a15e7413d9e', 'request_id': 'req-c6ef5a51-4456-4959-b033-852432b2ed78'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.050305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7e1d9b8ca5f4f5a95684a15e7413d9e', 'request_id': 'req-c6ef5a51-4456-4959-b033-852432b2ed78'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.058641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7e1d9b8ca5f4f5a95684a15e7413d9e', 'request_id': 'req-c6ef5a51-4456-4959-b033-852432b2ed78'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.070293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] The token cache is being invalidate because project a7e1d9b8ca5f4f5a95684a15e7413d9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:22.070582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.071256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.072738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 886/1768] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:21 2022] DELETE /identity/v3/projects/a7e1d9b8ca5f4f5a95684a15e7413d9e => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:22.077143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:22.079081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:22.099671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0616dca-adb0-4d38-89b2-66320783a3d1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:22.180491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:22.181371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:22.181742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:22.182087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] PATH_INFO: `/v3/users/35e573bb1e534703a44ac74c245a18f1` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:22.183327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35e573bb1e534703a44ac74c245a18f1)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:22.185322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:22.206286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:22.242600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35e573bb1e534703a44ac74c245a18f1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.249670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35e573bb1e534703a44ac74c245a18f1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.250484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35e573bb1e534703a44ac74c245a18f1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.251085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35e573bb1e534703a44ac74c245a18f1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.262303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f7887e34-3b8f-42a2-81f9-0d416990c667 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35e573bb1e534703a44ac74c245a18f1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.283331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 883/1769] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:22 2022] DELETE /identity/v3/users/35e573bb1e534703a44ac74c245a18f1 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:22.456917 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:22.458576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:22.472004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ef5a51-4456-4959-b033-852432b2ed78 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:22.538237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:22.538897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:22.539174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:22.539442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] PATH_INFO: `/v3/projects/fe0bf0dc097b43678c10294dfe4a39dd` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:22.540549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe0bf0dc097b43678c10294dfe4a39dd)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:22.542450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:22.593738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe0bf0dc097b43678c10294dfe4a39dd', 'request_id': 'req-a40886ae-6c21-4aeb-b7d9-9085207ce474'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.595141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:22.600680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe0bf0dc097b43678c10294dfe4a39dd', 'request_id': 'req-a40886ae-6c21-4aeb-b7d9-9085207ce474'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.613254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe0bf0dc097b43678c10294dfe4a39dd', 'request_id': 'req-a40886ae-6c21-4aeb-b7d9-9085207ce474'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.634996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] The token cache is being invalidate because project fe0bf0dc097b43678c10294dfe4a39dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:22.635274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.635939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a40886ae-6c21-4aeb-b7d9-9085207ce474 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:22.637746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 887/1770] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:22 2022] DELETE /identity/v3/projects/fe0bf0dc097b43678c10294dfe4a39dd => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:22.643781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feee722-d956-4bbc-b600-56a299c1d1b4 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:22.644071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feee722-d956-4bbc-b600-56a299c1d1b4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:22.644346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feee722-d956-4bbc-b600-56a299c1d1b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:22.664172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-6feee722-d956-4bbc-b600-56a299c1d1b4 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:22.665366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6feee722-d956-4bbc-b600-56a299c1d1b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:22.725109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 884/1771] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:22.732794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5e5bbe-42b0-4ee8-9ada-bc1438457133 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:22.733079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5e5bbe-42b0-4ee8-9ada-bc1438457133 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:22.733357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5e5bbe-42b0-4ee8-9ada-bc1438457133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:22.753929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3c5e5bbe-42b0-4ee8-9ada-bc1438457133 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:22.755179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3c5e5bbe-42b0-4ee8-9ada-bc1438457133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:22.763195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 888/1772] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:22.768070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feee722-d956-4bbc-b600-56a299c1d1b4 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:22.778019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feee722-d956-4bbc-b600-56a299c1d1b4 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:22.782839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:22.783451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:22.784430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:22.784430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:22.784765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:22.785110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:22.786722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:22.801516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 885/1773] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:22.805783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5e5bbe-42b0-4ee8-9ada-bc1438457133 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:22.815876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5e5bbe-42b0-4ee8-9ada-bc1438457133 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:22.824022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ffa15b-7050-4d74-afc2-30d079cf74d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:22.824658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ffa15b-7050-4d74-afc2-30d079cf74d5 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:22.824938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ffa15b-7050-4d74-afc2-30d079cf74d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:22.825211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ffa15b-7050-4d74-afc2-30d079cf74d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:22.825867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ffa15b-7050-4d74-afc2-30d079cf74d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:22.828459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ffa15b-7050-4d74-afc2-30d079cf74d5 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:22.858982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 889/1774] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:22 2022] POST /identity/v3/projects => generated 379 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:22.863311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:22.874595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779157eb-531b-48f3-a13f-4fa00e3dea95 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:22.936427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11a0143-514e-428b-8356-d5d2956583f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:22.937046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11a0143-514e-428b-8356-d5d2956583f8 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:22.937330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11a0143-514e-428b-8356-d5d2956583f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:22.937607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11a0143-514e-428b-8356-d5d2956583f8 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:22.938292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11a0143-514e-428b-8356-d5d2956583f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:22.940316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11a0143-514e-428b-8356-d5d2956583f8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:22.967057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 886/1775] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:22 2022] POST /identity/v3/users => generated 290 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:22.968459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ffa15b-7050-4d74-afc2-30d079cf74d5 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:22.979703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ffa15b-7050-4d74-afc2-30d079cf74d5 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:22.988151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d86256-bdf0-4938-8084-7173137f6e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:22.988919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d86256-bdf0-4938-8084-7173137f6e57 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:22.989514 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d86256-bdf0-4938-8084-7173137f6e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:22.989866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d86256-bdf0-4938-8084-7173137f6e57 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:22.990632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d86256-bdf0-4938-8084-7173137f6e57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:22.992398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d86256-bdf0-4938-8084-7173137f6e57 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.010160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 890/1776] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:23.011243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11a0143-514e-428b-8356-d5d2956583f8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.023337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11a0143-514e-428b-8356-d5d2956583f8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:23.029250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496586a2-5767-43c1-a02d-456f68a46bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:23.030211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496586a2-5767-43c1-a02d-456f68a46bda admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.030814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496586a2-5767-43c1-a02d-456f68a46bda admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.031233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496586a2-5767-43c1-a02d-456f68a46bda admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.032350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496586a2-5767-43c1-a02d-456f68a46bda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:23.034461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496586a2-5767-43c1-a02d-456f68a46bda admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.042087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:23.056605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 887/1777] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:23.061650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d86256-bdf0-4938-8084-7173137f6e57 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.088215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d86256-bdf0-4938-8084-7173137f6e57 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:23.094422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33d1a8f-1422-414b-9bf5-165be300b307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:23.095436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33d1a8f-1422-414b-9bf5-165be300b307 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.095717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33d1a8f-1422-414b-9bf5-165be300b307 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.095993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33d1a8f-1422-414b-9bf5-165be300b307 admin admin] PATH_INFO: `/v3/projects/9ea91eb1c3a5470c84c8887884a9552e/users/973bbb8c78a241d1aeac6557040fc36d/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.128344 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33d1a8f-1422-414b-9bf5-165be300b307 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ea91eb1c3a5470c84c8887884a9552e, user_id=973bbb8c78a241d1aeac6557040fc36d, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:23.132403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33d1a8f-1422-414b-9bf5-165be300b307 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.175301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 891/1778] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:23 2022] PUT /identity/v3/projects/9ea91eb1c3a5470c84c8887884a9552e/users/973bbb8c78a241d1aeac6557040fc36d/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:23.182057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f1def-23df-4a6b-ac42-01d591837201 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.183258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f1def-23df-4a6b-ac42-01d591837201 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.183848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f1def-23df-4a6b-ac42-01d591837201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.231824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-047f1def-23df-4a6b-ac42-01d591837201 None None] MFA Rules not processed for user `973bbb8c78a241d1aeac6557040fc36d`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:23.244854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-047f1def-23df-4a6b-ac42-01d591837201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:23.368605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 888/1779] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:23.369994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33d1a8f-1422-414b-9bf5-165be300b307 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.465477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33d1a8f-1422-414b-9bf5-165be300b307 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:23.470534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6eb8d-3402-49ed-81d4-37e919b1ba8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:23.471196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6eb8d-3402-49ed-81d4-37e919b1ba8e admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.471474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6eb8d-3402-49ed-81d4-37e919b1ba8e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.471747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6eb8d-3402-49ed-81d4-37e919b1ba8e admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.472396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e6eb8d-3402-49ed-81d4-37e919b1ba8e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:23.474326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e6eb8d-3402-49ed-81d4-37e919b1ba8e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.509003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 892/1780] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:23 2022] POST /identity/v3/projects => generated 379 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:23.510145 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047f1def-23df-4a6b-ac42-01d591837201 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.522119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047f1def-23df-4a6b-ac42-01d591837201 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:23.585333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a2f823-87e6-4105-8073-81dab5ae4cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:23.585963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a2f823-87e6-4105-8073-81dab5ae4cd3 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.586243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a2f823-87e6-4105-8073-81dab5ae4cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.586517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a2f823-87e6-4105-8073-81dab5ae4cd3 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.587209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a2f823-87e6-4105-8073-81dab5ae4cd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:23.589565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a2f823-87e6-4105-8073-81dab5ae4cd3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.614384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 889/1781] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:23 2022] POST /identity/v3/users => generated 290 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:23.618721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6eb8d-3402-49ed-81d4-37e919b1ba8e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.629714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6eb8d-3402-49ed-81d4-37e919b1ba8e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:23.635410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c88a5-59c9-4793-a64a-7c8bb782c6e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:23.636110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c88a5-59c9-4793-a64a-7c8bb782c6e7 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.636392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c88a5-59c9-4793-a64a-7c8bb782c6e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.636665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c88a5-59c9-4793-a64a-7c8bb782c6e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.637315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473c88a5-59c9-4793-a64a-7c8bb782c6e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:23.639523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473c88a5-59c9-4793-a64a-7c8bb782c6e7 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.654196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 893/1782] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:23.658863 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a2f823-87e6-4105-8073-81dab5ae4cd3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.670853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a2f823-87e6-4105-8073-81dab5ae4cd3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:23.680819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a91f797-87c7-4139-875e-36f01baa7846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:23.682065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91f797-87c7-4139-875e-36f01baa7846 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.682516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91f797-87c7-4139-875e-36f01baa7846 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.683306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91f797-87c7-4139-875e-36f01baa7846 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.684273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a91f797-87c7-4139-875e-36f01baa7846 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:23.686954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a91f797-87c7-4139-875e-36f01baa7846 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.702839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 890/1783] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:23 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:23.706970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c88a5-59c9-4793-a64a-7c8bb782c6e7 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.717634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c88a5-59c9-4793-a64a-7c8bb782c6e7 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:23.722965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4816cbd3-a3cc-47f8-af60-b6bdfa93ff0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:23.723706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4816cbd3-a3cc-47f8-af60-b6bdfa93ff0b admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.724054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4816cbd3-a3cc-47f8-af60-b6bdfa93ff0b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.724416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4816cbd3-a3cc-47f8-af60-b6bdfa93ff0b admin admin] PATH_INFO: `/v3/projects/4e187cae5dbf414cbae1cc15aaa9a503/users/d36daf335f9149549ef8024fd75e9d16/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.740191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4816cbd3-a3cc-47f8-af60-b6bdfa93ff0b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e187cae5dbf414cbae1cc15aaa9a503, user_id=d36daf335f9149549ef8024fd75e9d16, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:23.743025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4816cbd3-a3cc-47f8-af60-b6bdfa93ff0b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.757445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 894/1784] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:23 2022] PUT /identity/v3/projects/4e187cae5dbf414cbae1cc15aaa9a503/users/d36daf335f9149549ef8024fd75e9d16/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:23.768274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eb6862-69e7-4651-980d-fcffb91afda5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.768559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eb6862-69e7-4651-980d-fcffb91afda5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.768832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eb6862-69e7-4651-980d-fcffb91afda5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.798841 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-42eb6862-69e7-4651-980d-fcffb91afda5 None None] MFA Rules not processed for user `d36daf335f9149549ef8024fd75e9d16`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:23.800056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-42eb6862-69e7-4651-980d-fcffb91afda5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:23.859390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:23.865380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 891/1785] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:23.869685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4816cbd3-a3cc-47f8-af60-b6bdfa93ff0b admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.899063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:23.939419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4816cbd3-a3cc-47f8-af60-b6bdfa93ff0b admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:23.944469 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf64a4-f722-4a63-9847-6fb8f06faf13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:23.948947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf64a4-f722-4a63-9847-6fb8f06faf13 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:23.948947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf64a4-f722-4a63-9847-6fb8f06faf13 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:23.948947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf64a4-f722-4a63-9847-6fb8f06faf13 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:23.948947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bf64a4-f722-4a63-9847-6fb8f06faf13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:23.948947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bf64a4-f722-4a63-9847-6fb8f06faf13 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:23.977005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 895/1786] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:23 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:23.981171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eb6862-69e7-4651-980d-fcffb91afda5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:23.992939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eb6862-69e7-4651-980d-fcffb91afda5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:24.052825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0254d7ce-a864-4daf-803d-0a1309859fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:24.053441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254d7ce-a864-4daf-803d-0a1309859fcc admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:24.053722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254d7ce-a864-4daf-803d-0a1309859fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:24.053997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254d7ce-a864-4daf-803d-0a1309859fcc admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:24.054637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0254d7ce-a864-4daf-803d-0a1309859fcc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:24.056592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0254d7ce-a864-4daf-803d-0a1309859fcc admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:24.081866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 892/1787] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:23 2022] POST /identity/v3/users => generated 295 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:24.083722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf64a4-f722-4a63-9847-6fb8f06faf13 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:24.094673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf64a4-f722-4a63-9847-6fb8f06faf13 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:24.100603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9219c40a-9647-486d-ad6f-5a32db845214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:24.101213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9219c40a-9647-486d-ad6f-5a32db845214 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:24.101493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9219c40a-9647-486d-ad6f-5a32db845214 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:24.101765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9219c40a-9647-486d-ad6f-5a32db845214 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:24.102415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9219c40a-9647-486d-ad6f-5a32db845214 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:24.104052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9219c40a-9647-486d-ad6f-5a32db845214 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:24.115030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 896/1788] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:24.119258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0254d7ce-a864-4daf-803d-0a1309859fcc admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:24.130150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0254d7ce-a864-4daf-803d-0a1309859fcc admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:24.134990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f71bf76-694e-4d68-99d9-12a9b252a061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:24.135769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f71bf76-694e-4d68-99d9-12a9b252a061 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:24.136050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f71bf76-694e-4d68-99d9-12a9b252a061 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:24.136320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f71bf76-694e-4d68-99d9-12a9b252a061 admin admin] PATH_INFO: `/v3/projects/58efed42092f4a03a0185b841c5752e9/users/7f1c188f39334421a8507eb129fd9867/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:24.152060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f71bf76-694e-4d68-99d9-12a9b252a061 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58efed42092f4a03a0185b841c5752e9, user_id=7f1c188f39334421a8507eb129fd9867, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:24.154684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f71bf76-694e-4d68-99d9-12a9b252a061 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:24.168111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 893/1789] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:24 2022] PUT /identity/v3/projects/58efed42092f4a03a0185b841c5752e9/users/7f1c188f39334421a8507eb129fd9867/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:24.172017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9219c40a-9647-486d-ad6f-5a32db845214 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:24.182398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9219c40a-9647-486d-ad6f-5a32db845214 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:24.285029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4e1682-d027-4267-99f2-713098c6528e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:24.285938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4e1682-d027-4267-99f2-713098c6528e admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:24.286345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4e1682-d027-4267-99f2-713098c6528e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:24.286617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4e1682-d027-4267-99f2-713098c6528e admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:24.287304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4e1682-d027-4267-99f2-713098c6528e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:24.288987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4e1682-d027-4267-99f2-713098c6528e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:24.302867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 897/1790] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:24 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:24.304554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f71bf76-694e-4d68-99d9-12a9b252a061 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:24.324363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f71bf76-694e-4d68-99d9-12a9b252a061 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:24.332567 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:24.333325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:24.333601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:24.333870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] PATH_INFO: `/v3/projects/58efed42092f4a03a0185b841c5752e9/users/7f1c188f39334421a8507eb129fd9867/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:24.335920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] RBAC: Authorizing `identity:create_grant(project_id=58efed42092f4a03a0185b841c5752e9, user_id=7f1c188f39334421a8507eb129fd9867, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:24.338755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:24.350614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 894/1791] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:24 2022] PUT /identity/v3/projects/58efed42092f4a03a0185b841c5752e9/users/7f1c188f39334421a8507eb129fd9867/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:24.361444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caf8e0c-54cb-4d44-909b-587bec62b10b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:24.361727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caf8e0c-54cb-4d44-909b-587bec62b10b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:24.362012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caf8e0c-54cb-4d44-909b-587bec62b10b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:24.391257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2caf8e0c-54cb-4d44-909b-587bec62b10b None None] MFA Rules not processed for user `7f1c188f39334421a8507eb129fd9867`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:24.392562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2caf8e0c-54cb-4d44-909b-587bec62b10b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:24.466072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 898/1792] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:24 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:24.478958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:24.481045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:24.547704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d0ef2e-b780-4a3a-9b99-4bf8d0add18f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:24.551382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:24.551994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:24.552273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:24.552546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:24.562376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:24.563834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:24.573089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 895/1793] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:24 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:24.614994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caf8e0c-54cb-4d44-909b-587bec62b10b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:24.616364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2caf8e0c-54cb-4d44-909b-587bec62b10b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:24.629752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caf8e0c-54cb-4d44-909b-587bec62b10b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:24.634637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2c2a2b-fa2b-4692-a710-b56cab48dc1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:24.635308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2c2a2b-fa2b-4692-a710-b56cab48dc1f service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:24.635591 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2c2a2b-fa2b-4692-a710-b56cab48dc1f service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:24.635864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2c2a2b-fa2b-4692-a710-b56cab48dc1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:24.652791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:24.693542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2c2a2b-fa2b-4692-a710-b56cab48dc1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:24.695017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2c2a2b-fa2b-4692-a710-b56cab48dc1f service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:24.703834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 899/1794] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:24 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:31.227427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:31.230310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:31.303407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:31.318175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3649a7a-65d5-4047-9c6b-e6adf43d4f67 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:31.322764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:31.323445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:31.323752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:31.324027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] PATH_INFO: `/v3/users/3b0f415db6cc4ae6a4a686a53060fa99` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:31.325161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b0f415db6cc4ae6a4a686a53060fa99)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:31.327161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:31.372158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b0f415db6cc4ae6a4a686a53060fa99'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:31.379625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b0f415db6cc4ae6a4a686a53060fa99'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:31.380314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b0f415db6cc4ae6a4a686a53060fa99'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:31.381000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b0f415db6cc4ae6a4a686a53060fa99'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:31.388138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b0f415db6cc4ae6a4a686a53060fa99'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:31.405935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 896/1795] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:31 2022] DELETE /identity/v3/users/3b0f415db6cc4ae6a4a686a53060fa99 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:31.410383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710aaebb-2f86-4f0f-90cc-34f2873c1827 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:31.410876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710aaebb-2f86-4f0f-90cc-34f2873c1827 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:31.410965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710aaebb-2f86-4f0f-90cc-34f2873c1827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:31.431274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-710aaebb-2f86-4f0f-90cc-34f2873c1827 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:31.432710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-710aaebb-2f86-4f0f-90cc-34f2873c1827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:31.495327 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 900/1796] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:31.508076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:31.509664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:31.575977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97edbdff-ab88-49af-aa73-38c0446b7978 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:31.579615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:31.580224 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:31.580498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:31.580768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:31.590124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:31.592668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:31.604698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 897/1797] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:31.900887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710aaebb-2f86-4f0f-90cc-34f2873c1827 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:31.903318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-710aaebb-2f86-4f0f-90cc-34f2873c1827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:31.953376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:31.955018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:31.967144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710aaebb-2f86-4f0f-90cc-34f2873c1827 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:31.973322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:31.973969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:31.974249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:31.974523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] PATH_INFO: `/v3/projects/e2dc552429b044f7810faea6e1ebd40d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:31.975720 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2dc552429b044f7810faea6e1ebd40d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:31.977619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:31.978699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd086bed-e0df-4985-b665-436200bcc849 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:31.985258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:31.995186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:31.995483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:31.995897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] PATH_INFO: `/v3/users/973bbb8c78a241d1aeac6557040fc36d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:31.997262 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] RBAC: Authorizing `identity:delete_user(user_id=973bbb8c78a241d1aeac6557040fc36d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:31.999698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:32.041645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2dc552429b044f7810faea6e1ebd40d', 'request_id': 'req-6e013460-68f3-485d-9245-7db210a62feb'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.050155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2dc552429b044f7810faea6e1ebd40d', 'request_id': 'req-6e013460-68f3-485d-9245-7db210a62feb'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.058417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2dc552429b044f7810faea6e1ebd40d', 'request_id': 'req-6e013460-68f3-485d-9245-7db210a62feb'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.062471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '973bbb8c78a241d1aeac6557040fc36d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.070587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '973bbb8c78a241d1aeac6557040fc36d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.071669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '973bbb8c78a241d1aeac6557040fc36d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.072309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '973bbb8c78a241d1aeac6557040fc36d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.074083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] The token cache is being invalidate because project e2dc552429b044f7810faea6e1ebd40d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:32.074299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.074996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.086931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 901/1798] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:31 2022] DELETE /identity/v3/projects/e2dc552429b044f7810faea6e1ebd40d => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:32.086931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:32.086931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:32.092108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd5d0372-cb5f-4211-a767-ceb3f4d93947 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '973bbb8c78a241d1aeac6557040fc36d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.099812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:32.125575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 898/1799] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:31 2022] DELETE /identity/v3/users/973bbb8c78a241d1aeac6557040fc36d => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:32.130437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db10491-a484-4fa6-81dd-5a33456c9b1a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:32.130793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db10491-a484-4fa6-81dd-5a33456c9b1a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:32.131079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db10491-a484-4fa6-81dd-5a33456c9b1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:32.145961 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:32.152565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7db10491-a484-4fa6-81dd-5a33456c9b1a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:32.153836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7db10491-a484-4fa6-81dd-5a33456c9b1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:32.173078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e013460-68f3-485d-9245-7db210a62feb admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:32.230648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 899/1800] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:32.239963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:32.240618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:32.240901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:32.241066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db10491-a484-4fa6-81dd-5a33456c9b1a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:32.241179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] PATH_INFO: `/v3/users/0eea3d03955e4f4b8f2156113cfd7863` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:32.242314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] RBAC: Authorizing `identity:delete_user(user_id=0eea3d03955e4f4b8f2156113cfd7863)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:32.242419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7db10491-a484-4fa6-81dd-5a33456c9b1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:32.244311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:32.293095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0eea3d03955e4f4b8f2156113cfd7863'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.293947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0eea3d03955e4f4b8f2156113cfd7863'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.301394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0eea3d03955e4f4b8f2156113cfd7863'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.303357 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db10491-a484-4fa6-81dd-5a33456c9b1a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:32.307639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c0a1c4-b543-435a-bcba-ac402250d571 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:32.308310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c0a1c4-b543-435a-bcba-ac402250d571 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:32.308584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c0a1c4-b543-435a-bcba-ac402250d571 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:32.309057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c0a1c4-b543-435a-bcba-ac402250d571 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:32.311264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0eea3d03955e4f4b8f2156113cfd7863'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.322474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c0a1c4-b543-435a-bcba-ac402250d571 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:32.324456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c0a1c4-b543-435a-bcba-ac402250d571 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:32.324882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0eea3d03955e4f4b8f2156113cfd7863'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.333925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 902/1801] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:32 2022] DELETE /identity/v3/users/0eea3d03955e4f4b8f2156113cfd7863 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:32.338652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 900/1802] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:41:32.617352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:32.619234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:32.688310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a5c89e-a099-41cb-9d82-1e9966cfcd5a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:32.693134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:32.693739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:32.694017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:32.694313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] PATH_INFO: `/v3/projects/9ea91eb1c3a5470c84c8887884a9552e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:32.695455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ea91eb1c3a5470c84c8887884a9552e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:32.697353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:32.708716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:32.749736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ea91eb1c3a5470c84c8887884a9552e', 'request_id': 'req-235269a0-f609-4f33-94f8-4605d936f285'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.753131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c0a1c4-b543-435a-bcba-ac402250d571 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:32.756375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ea91eb1c3a5470c84c8887884a9552e', 'request_id': 'req-235269a0-f609-4f33-94f8-4605d936f285'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.764646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ea91eb1c3a5470c84c8887884a9552e', 'request_id': 'req-235269a0-f609-4f33-94f8-4605d936f285'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.764864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c0a1c4-b543-435a-bcba-ac402250d571 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:32.782372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] The token cache is being invalidate because project 9ea91eb1c3a5470c84c8887884a9552e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:32.782871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.784102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.786362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 903/1803] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:32 2022] DELETE /identity/v3/projects/9ea91eb1c3a5470c84c8887884a9552e => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:32.792256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:32.794263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:32.833193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:32.833834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:32.834169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:32.834446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] PATH_INFO: `/v3/projects/30e25f6c7ade474c91942916851c81bd` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:32.835605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] RBAC: Authorizing `identity:delete_project(project_id=30e25f6c7ade474c91942916851c81bd)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:32.837542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:32.870570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235269a0-f609-4f33-94f8-4605d936f285 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:32.873532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:32.876056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:32.876889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:32.877168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:32.877579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] PATH_INFO: `/v3/users/d36daf335f9149549ef8024fd75e9d16` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:32.878766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d36daf335f9149549ef8024fd75e9d16)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:32.880892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:32.886552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30e25f6c7ade474c91942916851c81bd', 'request_id': 'req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.895303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30e25f6c7ade474c91942916851c81bd', 'request_id': 'req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.907009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30e25f6c7ade474c91942916851c81bd', 'request_id': 'req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.924483 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] The token cache is being invalidate because project 30e25f6c7ade474c91942916851c81bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:32.924902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.926017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-0f8cf013-29f6-4b4d-adb6-5001393b6fbe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.928016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 901/1804] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:32 2022] DELETE /identity/v3/projects/30e25f6c7ade474c91942916851c81bd => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:32.932561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fccb31-f930-4116-8969-589335584a69 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:32.932854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fccb31-f930-4116-8969-589335584a69 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:32.933129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fccb31-f930-4116-8969-589335584a69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:32.938127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd36daf335f9149549ef8024fd75e9d16'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.938924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd36daf335f9149549ef8024fd75e9d16'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.947762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd36daf335f9149549ef8024fd75e9d16'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.956505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-81fccb31-f930-4116-8969-589335584a69 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:32.956973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd36daf335f9149549ef8024fd75e9d16'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.957927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-81fccb31-f930-4116-8969-589335584a69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:32.982000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1e12fcf0-957e-47f9-aebe-5fa43399ab0f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd36daf335f9149549ef8024fd75e9d16'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:32.991660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 904/1805] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:32 2022] DELETE /identity/v3/users/d36daf335f9149549ef8024fd75e9d16 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:33.031703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 902/1806] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:33.040418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f402dd3-f738-41d1-8422-f0fd35c6140c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.040706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f402dd3-f738-41d1-8422-f0fd35c6140c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.041009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f402dd3-f738-41d1-8422-f0fd35c6140c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.063425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3f402dd3-f738-41d1-8422-f0fd35c6140c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:33.064648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3f402dd3-f738-41d1-8422-f0fd35c6140c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:33.079116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:33.135610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 905/1807] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:33.143686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fccb31-f930-4116-8969-589335584a69 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.153726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fccb31-f930-4116-8969-589335584a69 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.163576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.163576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.163576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.163576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.163996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:33.163996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.163996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.176815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 903/1808] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:33.181291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f402dd3-f738-41d1-8422-f0fd35c6140c None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.193825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f402dd3-f738-41d1-8422-f0fd35c6140c None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.198932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.199560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.199871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.200147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.201386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.203525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.245121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 906/1809] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:33 2022] POST /identity/v3/projects => generated 389 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:33.249704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.261561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593554aa-7d8b-4024-8973-fd061b23d32a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.341277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266bb9d5-df67-4a5b-ba1d-16ad7fed2e12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.341778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.342125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266bb9d5-df67-4a5b-ba1d-16ad7fed2e12 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.342411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266bb9d5-df67-4a5b-ba1d-16ad7fed2e12 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.342790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266bb9d5-df67-4a5b-ba1d-16ad7fed2e12 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.343530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:33.343673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266bb9d5-df67-4a5b-ba1d-16ad7fed2e12 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.346047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266bb9d5-df67-4a5b-ba1d-16ad7fed2e12 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.359296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40ec9f6-1b66-4e09-ae29-a00a25831299 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.368680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.369438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.369793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.370146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] PATH_INFO: `/v3/projects/4e187cae5dbf414cbae1cc15aaa9a503` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.371623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e187cae5dbf414cbae1cc15aaa9a503)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.375033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.376477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 904/1810] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:33 2022] POST /identity/v3/users => generated 295 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:33.380388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266bb9d5-df67-4a5b-ba1d-16ad7fed2e12 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.392308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266bb9d5-df67-4a5b-ba1d-16ad7fed2e12 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.399018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dcefeb-3021-442d-a5d8-4ceb920d8f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.399721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dcefeb-3021-442d-a5d8-4ceb920d8f7e admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.400071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dcefeb-3021-442d-a5d8-4ceb920d8f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.400413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dcefeb-3021-442d-a5d8-4ceb920d8f7e admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.401170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dcefeb-3021-442d-a5d8-4ceb920d8f7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.402906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dcefeb-3021-442d-a5d8-4ceb920d8f7e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.414781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 905/1811] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:33 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:33.418757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dcefeb-3021-442d-a5d8-4ceb920d8f7e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.427847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e187cae5dbf414cbae1cc15aaa9a503', 'request_id': 'req-542e3d45-3c65-4908-b4b8-3659986db3de'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.429399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dcefeb-3021-442d-a5d8-4ceb920d8f7e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.434166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e187cae5dbf414cbae1cc15aaa9a503', 'request_id': 'req-542e3d45-3c65-4908-b4b8-3659986db3de'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.443135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e187cae5dbf414cbae1cc15aaa9a503', 'request_id': 'req-542e3d45-3c65-4908-b4b8-3659986db3de'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.454506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] The token cache is being invalidate because project 4e187cae5dbf414cbae1cc15aaa9a503 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:33.455032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.455785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.457329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 907/1812] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:33 2022] DELETE /identity/v3/projects/4e187cae5dbf414cbae1cc15aaa9a503 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:33.457692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:33.463690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.465569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:33.496425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.497041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.497325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.498289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.498983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.500595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.510589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 906/1813] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:33 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:33.514689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.516396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:33.528752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7245b3b-1475-4a19-b024-d168e7363200 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.533609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbe13af-7000-412a-ae08-f908bb80b13d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.534362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe13af-7000-412a-ae08-f908bb80b13d admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.534646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe13af-7000-412a-ae08-f908bb80b13d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.534959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe13af-7000-412a-ae08-f908bb80b13d admin admin] PATH_INFO: `/v3/projects/7110fea89c1f4bfabbefa7d793490c3f/users/0bd9802e1d7f4c58a27de9678acaec83/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.537580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e3d45-3c65-4908-b4b8-3659986db3de admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.542514 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.543186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.543473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.543744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] PATH_INFO: `/v3/users/7f1c188f39334421a8507eb129fd9867` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.545400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f1c188f39334421a8507eb129fd9867)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.547394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.551045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbe13af-7000-412a-ae08-f908bb80b13d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7110fea89c1f4bfabbefa7d793490c3f, user_id=0bd9802e1d7f4c58a27de9678acaec83, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.553975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbe13af-7000-412a-ae08-f908bb80b13d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.571662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 907/1814] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:33 2022] PUT /identity/v3/projects/7110fea89c1f4bfabbefa7d793490c3f/users/0bd9802e1d7f4c58a27de9678acaec83/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:33.576042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:33.583906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d346ded0-0d57-481a-86d0-d45cc5867b01 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.584209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d346ded0-0d57-481a-86d0-d45cc5867b01 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.584541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d346ded0-0d57-481a-86d0-d45cc5867b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.608390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f1c188f39334421a8507eb129fd9867'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.608996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f1c188f39334421a8507eb129fd9867'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.616988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f1c188f39334421a8507eb129fd9867'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.622104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d346ded0-0d57-481a-86d0-d45cc5867b01 None None] MFA Rules not processed for user `0bd9802e1d7f4c58a27de9678acaec83`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:33.624375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d346ded0-0d57-481a-86d0-d45cc5867b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:33.626475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f1c188f39334421a8507eb129fd9867'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.639808 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f1c188f39334421a8507eb129fd9867'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:33.647432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 908/1815] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:33 2022] DELETE /identity/v3/users/7f1c188f39334421a8507eb129fd9867 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:33.700387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 908/1816] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:33 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:33.705320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.706927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:33.795158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f62a23-d03f-4a77-92b5-58198b684eab admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.801136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.801889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.802252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.802598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.803405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.805466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:33.836625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 909/1817] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:33 2022] POST /identity/v3/projects => generated 389 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:33.840839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d346ded0-0d57-481a-86d0-d45cc5867b01 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.906685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d346ded0-0d57-481a-86d0-d45cc5867b01 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:33.940579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:33.949670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:33.951656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:33.975978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86446714-a07b-432e-8fb5-78cd41f933ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:33.976596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86446714-a07b-432e-8fb5-78cd41f933ac admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:33.976885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86446714-a07b-432e-8fb5-78cd41f933ac admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:33.977492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86446714-a07b-432e-8fb5-78cd41f933ac admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:33.977811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86446714-a07b-432e-8fb5-78cd41f933ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:33.980560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86446714-a07b-432e-8fb5-78cd41f933ac admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:34.042044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 909/1818] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:33 2022] POST /identity/v3/users => generated 301 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:34.048186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86446714-a07b-432e-8fb5-78cd41f933ac admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:34.069650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86446714-a07b-432e-8fb5-78cd41f933ac admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:34.083704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83eeaab-de5f-4e0c-b5bf-060337b4a9ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:34.084100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83eeaab-de5f-4e0c-b5bf-060337b4a9ff admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.084387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83eeaab-de5f-4e0c-b5bf-060337b4a9ff admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.084663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83eeaab-de5f-4e0c-b5bf-060337b4a9ff admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.085507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83eeaab-de5f-4e0c-b5bf-060337b4a9ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:34.088856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83eeaab-de5f-4e0c-b5bf-060337b4a9ff admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:34.119154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c77816-b883-41a2-b441-592649452c8b admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:34.127400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 910/1819] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:34 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:34.128400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83eeaab-de5f-4e0c-b5bf-060337b4a9ff admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:34.143149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:34.147661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.147950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.148228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] PATH_INFO: `/v3/projects/58efed42092f4a03a0185b841c5752e9` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.150703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83eeaab-de5f-4e0c-b5bf-060337b4a9ff admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:34.153597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58efed42092f4a03a0185b841c5752e9)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:34.171609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:34.173592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418bba5-247f-4914-8647-f5dcc2a3429c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:34.174505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418bba5-247f-4914-8647-f5dcc2a3429c admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.174964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418bba5-247f-4914-8647-f5dcc2a3429c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.175399 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418bba5-247f-4914-8647-f5dcc2a3429c admin admin] PATH_INFO: `/v3/projects/f779e7fcf7cf4e28aed9bcd920c0ea4e/users/bbba2df21a8e49ab803aa22b919a535d/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.212446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7418bba5-247f-4914-8647-f5dcc2a3429c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f779e7fcf7cf4e28aed9bcd920c0ea4e, user_id=bbba2df21a8e49ab803aa22b919a535d, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:34.217586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7418bba5-247f-4914-8647-f5dcc2a3429c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:34.240547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=8,keystone:SELECT=90,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:34.252150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 911/1820] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:34 2022] PUT /identity/v3/projects/f779e7fcf7cf4e28aed9bcd920c0ea4e/users/bbba2df21a8e49ab803aa22b919a535d/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:34.256678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418bba5-247f-4914-8647-f5dcc2a3429c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:34.288076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58efed42092f4a03a0185b841c5752e9', 'request_id': 'req-d5982f21-114d-44e9-b7d6-11a854f8d8b5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:34.297278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58efed42092f4a03a0185b841c5752e9', 'request_id': 'req-d5982f21-114d-44e9-b7d6-11a854f8d8b5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:34.318911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58efed42092f4a03a0185b841c5752e9', 'request_id': 'req-d5982f21-114d-44e9-b7d6-11a854f8d8b5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:34.340519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] The token cache is being invalidate because project 58efed42092f4a03a0185b841c5752e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:34.340716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:34.341395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d5982f21-114d-44e9-b7d6-11a854f8d8b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:34.349385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 910/1821] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:33 2022] DELETE /identity/v3/projects/58efed42092f4a03a0185b841c5752e9 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:34.350908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb3554-2e00-49a1-95d7-420f08b36d0c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.351226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb3554-2e00-49a1-95d7-420f08b36d0c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.351510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb3554-2e00-49a1-95d7-420f08b36d0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.392761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-4bcb3554-2e00-49a1-95d7-420f08b36d0c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:34.395054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418bba5-247f-4914-8647-f5dcc2a3429c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:34.398912 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4bcb3554-2e00-49a1-95d7-420f08b36d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:34.553410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 911/1822] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:34.563424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a824c691-702a-4f42-9fab-153da3abd8d7 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.563716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a824c691-702a-4f42-9fab-153da3abd8d7 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.563996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a824c691-702a-4f42-9fab-153da3abd8d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.566100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:34.567131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.567736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.568199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.569197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:34.571517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:34.588670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 912/1823] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:34 2022] GET /identity/v3/roles => generated 1420 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:34.593115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:34.596806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:34.604257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a824c691-702a-4f42-9fab-153da3abd8d7 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:34.605361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a824c691-702a-4f42-9fab-153da3abd8d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:34.622077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443d8559-b937-4464-8778-2d5070118873 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:34.632498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 912/1824] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:34.633854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a824c691-702a-4f42-9fab-153da3abd8d7 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:34.641701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3185010-bac0-48e3-9da7-0fd7e67c17d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:34.644546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3185010-bac0-48e3-9da7-0fd7e67c17d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.645675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3185010-bac0-48e3-9da7-0fd7e67c17d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.646445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3185010-bac0-48e3-9da7-0fd7e67c17d7 admin admin] PATH_INFO: `/v3/projects/f779e7fcf7cf4e28aed9bcd920c0ea4e/users/bbba2df21a8e49ab803aa22b919a535d/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.651693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a824c691-702a-4f42-9fab-153da3abd8d7 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:34.654886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3185010-bac0-48e3-9da7-0fd7e67c17d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f779e7fcf7cf4e28aed9bcd920c0ea4e, user_id=bbba2df21a8e49ab803aa22b919a535d, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:34.660304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3185010-bac0-48e3-9da7-0fd7e67c17d7 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:34.667028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:34.667664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.668002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.668283 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.668963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:34.669320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:34.671662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:34.679085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 913/1825] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:34 2022] PUT /identity/v3/projects/f779e7fcf7cf4e28aed9bcd920c0ea4e/users/bbba2df21a8e49ab803aa22b919a535d/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:34.690333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 913/1826] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:34.691858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:34.692464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e02748-0b92-4d89-9303-208aa74607e7 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.692807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e02748-0b92-4d89-9303-208aa74607e7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.693088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e02748-0b92-4d89-9303-208aa74607e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.713503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c8983-ee7b-46dc-9790-633964851775 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:34.744661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-10e02748-0b92-4d89-9303-208aa74607e7 None None] MFA Rules not processed for user `bbba2df21a8e49ab803aa22b919a535d`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:34.746571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-10e02748-0b92-4d89-9303-208aa74607e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:34.758192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:34.857351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:34.858215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.858506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.858818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.860311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:34.862538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:34.895830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 914/1827] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:34 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:34.903451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 914/1828] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:34 2022] POST /identity/v3/projects => generated 367 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:34.907543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e02748-0b92-4d89-9303-208aa74607e7 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:34.907870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:34.909866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:34.921523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e02748-0b92-4d89-9303-208aa74607e7 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:34.945095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:34.996707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccf6ce5-e496-46ab-9bf0-8c58709568a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:34.997481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccf6ce5-e496-46ab-9bf0-8c58709568a1 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:34.997766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccf6ce5-e496-46ab-9bf0-8c58709568a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:34.998176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccf6ce5-e496-46ab-9bf0-8c58709568a1 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:34.998879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccf6ce5-e496-46ab-9bf0-8c58709568a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:34.998987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4900019c-c52d-4b54-ad05-7fc3a406e30d admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:35.000810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccf6ce5-e496-46ab-9bf0-8c58709568a1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:35.003074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2255fdc-38c6-4155-9257-dcf9ea285a3d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:35.003707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2255fdc-38c6-4155-9257-dcf9ea285a3d service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:35.004218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2255fdc-38c6-4155-9257-dcf9ea285a3d service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:35.004499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2255fdc-38c6-4155-9257-dcf9ea285a3d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:35.036514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 915/1829] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:34 2022] POST /identity/v3/users => generated 284 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:35.048612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccf6ce5-e496-46ab-9bf0-8c58709568a1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:35.065976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccf6ce5-e496-46ab-9bf0-8c58709568a1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:35.072800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc39ee89-49dc-4244-b4df-0df79e6f93a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:35.073699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc39ee89-49dc-4244-b4df-0df79e6f93a0 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:35.074218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc39ee89-49dc-4244-b4df-0df79e6f93a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:35.074607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc39ee89-49dc-4244-b4df-0df79e6f93a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:35.075440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc39ee89-49dc-4244-b4df-0df79e6f93a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:35.078657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc39ee89-49dc-4244-b4df-0df79e6f93a0 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:35.103958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 916/1830] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:35 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:35.105644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc39ee89-49dc-4244-b4df-0df79e6f93a0 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:35.122844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc39ee89-49dc-4244-b4df-0df79e6f93a0 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:35.127830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed2abab-e46a-4b8a-a44b-c636faa4bb2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:35.128521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed2abab-e46a-4b8a-a44b-c636faa4bb2e admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:35.128736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed2abab-e46a-4b8a-a44b-c636faa4bb2e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:35.129200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed2abab-e46a-4b8a-a44b-c636faa4bb2e admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:35.129615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2255fdc-38c6-4155-9257-dcf9ea285a3d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:35.129914 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed2abab-e46a-4b8a-a44b-c636faa4bb2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:35.131942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2255fdc-38c6-4155-9257-dcf9ea285a3d service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:35.132546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed2abab-e46a-4b8a-a44b-c636faa4bb2e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:35.149730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 917/1831] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:35 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:35.149910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 915/1832] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:34 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:35.151235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed2abab-e46a-4b8a-a44b-c636faa4bb2e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:35.166242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed2abab-e46a-4b8a-a44b-c636faa4bb2e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:35.187882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84acc3d-8427-4342-bd33-5fa6ce79cb65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:35.188652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84acc3d-8427-4342-bd33-5fa6ce79cb65 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:35.188942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84acc3d-8427-4342-bd33-5fa6ce79cb65 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:35.189364 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84acc3d-8427-4342-bd33-5fa6ce79cb65 admin admin] PATH_INFO: `/v3/projects/2970058e05374180a32e27f8496bc290/users/4a2356f71a58407e95d4495fb7068c27/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:35.242169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84acc3d-8427-4342-bd33-5fa6ce79cb65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2970058e05374180a32e27f8496bc290, user_id=4a2356f71a58407e95d4495fb7068c27, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:35.248642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84acc3d-8427-4342-bd33-5fa6ce79cb65 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:35.294441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 918/1833] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:35 2022] PUT /identity/v3/projects/2970058e05374180a32e27f8496bc290/users/4a2356f71a58407e95d4495fb7068c27/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:35.298137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d105aa7c-0d5b-4c52-8a5d-71033a49f46c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:35.298648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d105aa7c-0d5b-4c52-8a5d-71033a49f46c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:35.299063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d105aa7c-0d5b-4c52-8a5d-71033a49f46c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:35.375411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d105aa7c-0d5b-4c52-8a5d-71033a49f46c None None] MFA Rules not processed for user `4a2356f71a58407e95d4495fb7068c27`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:35.376666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d105aa7c-0d5b-4c52-8a5d-71033a49f46c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:35.465845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84acc3d-8427-4342-bd33-5fa6ce79cb65 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:35.520882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 916/1834] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:35 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:35.525426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d105aa7c-0d5b-4c52-8a5d-71033a49f46c None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:35.595140 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:35.633737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84acc3d-8427-4342-bd33-5fa6ce79cb65 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:35.651194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:35.651434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:35.651643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:35.653749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] PATH_INFO: `/v3/users/0bd9802e1d7f4c58a27de9678acaec83` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:35.656594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=0bd9802e1d7f4c58a27de9678acaec83)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:35.661696 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:35.678395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d105aa7c-0d5b-4c52-8a5d-71033a49f46c None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:35.691773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4327a8-8e08-414d-a8a5-cfef474686cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:35.692616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4327a8-8e08-414d-a8a5-cfef474686cd admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:35.693004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4327a8-8e08-414d-a8a5-cfef474686cd admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:35.693382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4327a8-8e08-414d-a8a5-cfef474686cd admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:35.694319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4327a8-8e08-414d-a8a5-cfef474686cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:35.696963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4327a8-8e08-414d-a8a5-cfef474686cd admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:35.702711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:35.768044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 917/1835] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:35 2022] POST /identity/v3/projects => generated 365 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:35.772208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4327a8-8e08-414d-a8a5-cfef474686cd admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:35.820942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0bd9802e1d7f4c58a27de9678acaec83'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:35.858744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0bd9802e1d7f4c58a27de9678acaec83'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:35.860217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0bd9802e1d7f4c58a27de9678acaec83'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:35.863143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0bd9802e1d7f4c58a27de9678acaec83'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:35.878161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-40fdb3af-be10-4eae-a9b5-0a4c47d59b3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0bd9802e1d7f4c58a27de9678acaec83'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:35.922382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 919/1836] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:35 2022] DELETE /identity/v3/users/0bd9802e1d7f4c58a27de9678acaec83 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:35.924281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4327a8-8e08-414d-a8a5-cfef474686cd admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:35.928670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab1924-2696-447e-baa7-1bf68b8d4a98 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:35.929110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab1924-2696-447e-baa7-1bf68b8d4a98 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:35.929949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab1924-2696-447e-baa7-1bf68b8d4a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:35.963407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3aab1924-2696-447e-baa7-1bf68b8d4a98 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:35.965880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3aab1924-2696-447e-baa7-1bf68b8d4a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:36.079160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:36.080473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.092043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 920/1837] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:36.095645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.095645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.095645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:36.096092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:36.108855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aab1924-2696-447e-baa7-1bf68b8d4a98 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:36.111630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3aab1924-2696-447e-baa7-1bf68b8d4a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:36.143131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 918/1838] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:35 2022] POST /identity/v3/users => generated 289 bytes in 369 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:36.147569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:36.150249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:36.171118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2667212a-e93a-476c-a67a-dcfcfdd7ca8b admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:36.176630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e73e1e7-1595-4d32-b55d-179e6cb86786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:36.177369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73e1e7-1595-4d32-b55d-179e6cb86786 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.177715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73e1e7-1595-4d32-b55d-179e6cb86786 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.177997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73e1e7-1595-4d32-b55d-179e6cb86786 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.178707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e73e1e7-1595-4d32-b55d-179e6cb86786 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:36.180555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e73e1e7-1595-4d32-b55d-179e6cb86786 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:36.195860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 919/1839] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:36 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:36.199444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e73e1e7-1595-4d32-b55d-179e6cb86786 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:36.211568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e73e1e7-1595-4d32-b55d-179e6cb86786 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:36.217611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bde5f4-e7d0-4f53-8ad6-eff58854f643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:36.218167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bde5f4-e7d0-4f53-8ad6-eff58854f643 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.218452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bde5f4-e7d0-4f53-8ad6-eff58854f643 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.218725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bde5f4-e7d0-4f53-8ad6-eff58854f643 admin admin] PATH_INFO: `/v3/projects/64be32f0d7d4445aa5640f8ff1d9886f/users/560baf4e37034dbbbfa8421074f8c8b2/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.228404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aab1924-2696-447e-baa7-1bf68b8d4a98 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:36.236533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bde5f4-e7d0-4f53-8ad6-eff58854f643 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64be32f0d7d4445aa5640f8ff1d9886f, user_id=560baf4e37034dbbbfa8421074f8c8b2, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:36.237475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c871d8-c7a2-4494-849b-b2d3a911c3b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:36.238484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c871d8-c7a2-4494-849b-b2d3a911c3b6 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.239283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c871d8-c7a2-4494-849b-b2d3a911c3b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.239401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bde5f4-e7d0-4f53-8ad6-eff58854f643 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:36.239527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c871d8-c7a2-4494-849b-b2d3a911c3b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.257933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c871d8-c7a2-4494-849b-b2d3a911c3b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:36.264600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 920/1840] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:36 2022] PUT /identity/v3/projects/64be32f0d7d4445aa5640f8ff1d9886f/users/560baf4e37034dbbbfa8421074f8c8b2/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:36.264787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c871d8-c7a2-4494-849b-b2d3a911c3b6 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:36.266774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bde5f4-e7d0-4f53-8ad6-eff58854f643 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:36.289337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 921/1841] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:41:36.332305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:36.352044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bde5f4-e7d0-4f53-8ad6-eff58854f643 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:36.370387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52954d00-6e8d-4e9b-815a-d9280a0b2cd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:36.373279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52954d00-6e8d-4e9b-815a-d9280a0b2cd9 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.375944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52954d00-6e8d-4e9b-815a-d9280a0b2cd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.376422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52954d00-6e8d-4e9b-815a-d9280a0b2cd9 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.377320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52954d00-6e8d-4e9b-815a-d9280a0b2cd9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:36.381820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52954d00-6e8d-4e9b-815a-d9280a0b2cd9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:36.423784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 921/1842] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:36 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:36.424869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c871d8-c7a2-4494-849b-b2d3a911c3b6 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:36.436043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4cb79-2423-43ed-9363-45f84a9c1a9a None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.436525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4cb79-2423-43ed-9363-45f84a9c1a9a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.436806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4cb79-2423-43ed-9363-45f84a9c1a9a None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.441037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 922/1843] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:41:36 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:41:36.448951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f423abea-abf4-47c6-8819-ea13761737b0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.449235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f423abea-abf4-47c6-8819-ea13761737b0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.449510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f423abea-abf4-47c6-8819-ea13761737b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.475688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c871d8-c7a2-4494-849b-b2d3a911c3b6 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:36.490792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb7e5e7-16c0-4dd3-9d85-0164986fc322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:36.508293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f423abea-abf4-47c6-8819-ea13761737b0 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:36.511290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f423abea-abf4-47c6-8819-ea13761737b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:36.512731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7e5e7-16c0-4dd3-9d85-0164986fc322 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.513250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7e5e7-16c0-4dd3-9d85-0164986fc322 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.513898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7e5e7-16c0-4dd3-9d85-0164986fc322 admin admin] PATH_INFO: `/v3/projects/64be32f0d7d4445aa5640f8ff1d9886f/users/560baf4e37034dbbbfa8421074f8c8b2/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.524398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb7e5e7-16c0-4dd3-9d85-0164986fc322 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64be32f0d7d4445aa5640f8ff1d9886f, user_id=560baf4e37034dbbbfa8421074f8c8b2, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:36.528077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb7e5e7-16c0-4dd3-9d85-0164986fc322 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:36.545210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:36.565429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 922/1844] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:36 2022] PUT /identity/v3/projects/64be32f0d7d4445aa5640f8ff1d9886f/users/560baf4e37034dbbbfa8421074f8c8b2/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:36.580928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ad29c0-695e-47b4-a70f-c4614b22c887 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.581582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ad29c0-695e-47b4-a70f-c4614b22c887 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.582010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ad29c0-695e-47b4-a70f-c4614b22c887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.680131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-91ad29c0-695e-47b4-a70f-c4614b22c887 None None] MFA Rules not processed for user `560baf4e37034dbbbfa8421074f8c8b2`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:36.681835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-91ad29c0-695e-47b4-a70f-c4614b22c887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:36.684152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 923/1845] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:41:36 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:36.755286 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f423abea-abf4-47c6-8819-ea13761737b0 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:36.756901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f423abea-abf4-47c6-8819-ea13761737b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:36.780945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f423abea-abf4-47c6-8819-ea13761737b0 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:36.914440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 923/1846] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:36 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 336 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:36.918438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ad29c0-695e-47b4-a70f-c4614b22c887 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:36.920904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-91ad29c0-695e-47b4-a70f-c4614b22c887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:36.959000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592d7289-96c5-44aa-8c8b-cc2d6ade3559 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:36.960131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592d7289-96c5-44aa-8c8b-cc2d6ade3559 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:36.960580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592d7289-96c5-44aa-8c8b-cc2d6ade3559 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:36.961015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592d7289-96c5-44aa-8c8b-cc2d6ade3559 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:36.982300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592d7289-96c5-44aa-8c8b-cc2d6ade3559 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:36.987243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592d7289-96c5-44aa-8c8b-cc2d6ade3559 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:37.040414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 924/1847] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:41:37.110783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ad29c0-695e-47b4-a70f-c4614b22c887 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:37.123315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:37.125062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:37.125383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:37.128411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:37.152481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:37.154461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:37.181286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 924/1848] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:36 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:41:37.345919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592d7289-96c5-44aa-8c8b-cc2d6ade3559 service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:37.357483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592d7289-96c5-44aa-8c8b-cc2d6ade3559 service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:37.362010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:37.362723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:37.363135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:37.363473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:37.366390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:37.455886 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:37.457552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:37.462367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:37.474303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 925/1849] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:37 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:38.176879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:38.178979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:38.280751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b738e1-81e5-406a-a2aa-e53033b368ef service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:38.285771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:38.286615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:38.287109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:38.287575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] PATH_INFO: `/v3/projects/7110fea89c1f4bfabbefa7d793490c3f` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:38.289335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] RBAC: Authorizing `identity:delete_project(project_id=7110fea89c1f4bfabbefa7d793490c3f)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:38.291520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:38.337728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:38.368235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7110fea89c1f4bfabbefa7d793490c3f', 'request_id': 'req-6abc08ba-1416-4211-81e7-1ab4a46b709f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.375778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7110fea89c1f4bfabbefa7d793490c3f', 'request_id': 'req-6abc08ba-1416-4211-81e7-1ab4a46b709f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.383575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7110fea89c1f4bfabbefa7d793490c3f', 'request_id': 'req-6abc08ba-1416-4211-81e7-1ab4a46b709f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.400336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] The token cache is being invalidate because project 7110fea89c1f4bfabbefa7d793490c3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:38.400626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.401275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6abc08ba-1416-4211-81e7-1ab4a46b709f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.403040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 925/1850] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:38 2022] DELETE /identity/v3/projects/7110fea89c1f4bfabbefa7d793490c3f => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:38.407376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:38.421532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7980da96-330c-4f62-80d6-caaaaddae854 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:38.498187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:38.498902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:38.499187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:38.499462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] PATH_INFO: `/v3/users/bbba2df21a8e49ab803aa22b919a535d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:38.500677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbba2df21a8e49ab803aa22b919a535d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:38.503289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:38.562149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f76dd5b-3c17-438b-a6ff-d040e16bddfa None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:38.562845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f76dd5b-3c17-438b-a6ff-d040e16bddfa None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:38.563520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f76dd5b-3c17-438b-a6ff-d040e16bddfa None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:38.565506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 926/1851] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:41:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 29 22:41:38.569551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbba2df21a8e49ab803aa22b919a535d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.570132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbba2df21a8e49ab803aa22b919a535d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.579015 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bd674-0e3a-4514-8d7c-a677e7fc4d54 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:38.579307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bd674-0e3a-4514-8d7c-a677e7fc4d54 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:38.579587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5bd674-0e3a-4514-8d7c-a677e7fc4d54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:38.583474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbba2df21a8e49ab803aa22b919a535d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.592330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbba2df21a8e49ab803aa22b919a535d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.602173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7a5bd674-0e3a-4514-8d7c-a677e7fc4d54 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:38.603494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7a5bd674-0e3a-4514-8d7c-a677e7fc4d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:38.610272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbba2df21a8e49ab803aa22b919a535d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:38.620627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 926/1852] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:38 2022] DELETE /identity/v3/users/bbba2df21a8e49ab803aa22b919a535d => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:38.693123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 927/1853] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:41:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:38.706629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:38.708305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:38.817425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445f2974-d833-479c-8033-c26b7ef693ed admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:38.824463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a01c2b-bda0-4288-9059-2da307367d8b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:38.825961 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a01c2b-bda0-4288-9059-2da307367d8b service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:38.826312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a01c2b-bda0-4288-9059-2da307367d8b service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:38.826675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a01c2b-bda0-4288-9059-2da307367d8b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:38.836800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a01c2b-bda0-4288-9059-2da307367d8b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:38.838676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a01c2b-bda0-4288-9059-2da307367d8b service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:38.852023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 927/1854] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:38.945197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5bd674-0e3a-4514-8d7c-a677e7fc4d54 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:38.946938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7a5bd674-0e3a-4514-8d7c-a677e7fc4d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:38.961859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5bd674-0e3a-4514-8d7c-a677e7fc4d54 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.054758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.055426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.056118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.056118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] PATH_INFO: `/v3/projects/f779e7fcf7cf4e28aed9bcd920c0ea4e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.057141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f779e7fcf7cf4e28aed9bcd920c0ea4e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.059145 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.116672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f779e7fcf7cf4e28aed9bcd920c0ea4e', 'request_id': 'req-bef18e2b-5263-4bbe-b355-780b24361f00'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:39.120003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:39.128642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f779e7fcf7cf4e28aed9bcd920c0ea4e', 'request_id': 'req-bef18e2b-5263-4bbe-b355-780b24361f00'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:39.137683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f779e7fcf7cf4e28aed9bcd920c0ea4e', 'request_id': 'req-bef18e2b-5263-4bbe-b355-780b24361f00'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:39.165088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] The token cache is being invalidate because project f779e7fcf7cf4e28aed9bcd920c0ea4e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:39.165387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:39.166432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bef18e2b-5263-4bbe-b355-780b24361f00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:39.168031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 928/1855] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:38 2022] DELETE /identity/v3/projects/f779e7fcf7cf4e28aed9bcd920c0ea4e => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:39.173031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f90caa1-b1bf-43f5-b48d-1bab9d1389af None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.173551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f90caa1-b1bf-43f5-b48d-1bab9d1389af None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.173757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f90caa1-b1bf-43f5-b48d-1bab9d1389af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.181312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:39.196109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-0f90caa1-b1bf-43f5-b48d-1bab9d1389af None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:39.197314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0f90caa1-b1bf-43f5-b48d-1bab9d1389af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:39.266084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 928/1856] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:39.275177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb097782-9eec-4c6f-99eb-379f9b531cdb None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.275699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb097782-9eec-4c6f-99eb-379f9b531cdb None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.275779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb097782-9eec-4c6f-99eb-379f9b531cdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.296097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-fb097782-9eec-4c6f-99eb-379f9b531cdb None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:39.297323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-fb097782-9eec-4c6f-99eb-379f9b531cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:39.305608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 929/1857] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:39.310310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f90caa1-b1bf-43f5-b48d-1bab9d1389af None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.320146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f90caa1-b1bf-43f5-b48d-1bab9d1389af None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.324703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.325307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.326204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.326476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.327159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:39.327463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.329058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.340941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 929/1858] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:39.345730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb097782-9eec-4c6f-99eb-379f9b531cdb None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.355129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb097782-9eec-4c6f-99eb-379f9b531cdb None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.360497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db43e8dd-469f-45f4-9b44-6cf78b26f673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.361103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db43e8dd-469f-45f4-9b44-6cf78b26f673 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.361381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db43e8dd-469f-45f4-9b44-6cf78b26f673 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.361654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db43e8dd-469f-45f4-9b44-6cf78b26f673 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.362309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db43e8dd-469f-45f4-9b44-6cf78b26f673 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.364404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db43e8dd-469f-45f4-9b44-6cf78b26f673 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.392772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 930/1859] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:39 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:39.397266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.408301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a0a4f1-4b98-4f1b-8802-3996d30e21ae admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.465101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dade4233-7aeb-4214-af56-b2b396f27b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.465696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade4233-7aeb-4214-af56-b2b396f27b8a admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.466011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade4233-7aeb-4214-af56-b2b396f27b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.466820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dade4233-7aeb-4214-af56-b2b396f27b8a admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.467473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dade4233-7aeb-4214-af56-b2b396f27b8a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.469401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dade4233-7aeb-4214-af56-b2b396f27b8a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.491924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 930/1860] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:39 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:39.495821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db43e8dd-469f-45f4-9b44-6cf78b26f673 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.506088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db43e8dd-469f-45f4-9b44-6cf78b26f673 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.510708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d133b08c-ab1b-4ad0-96fb-5b0d842c999d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.511354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d133b08c-ab1b-4ad0-96fb-5b0d842c999d admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.511647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d133b08c-ab1b-4ad0-96fb-5b0d842c999d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.511912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d133b08c-ab1b-4ad0-96fb-5b0d842c999d admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.512564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d133b08c-ab1b-4ad0-96fb-5b0d842c999d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.514717 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d133b08c-ab1b-4ad0-96fb-5b0d842c999d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.525175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 931/1861] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:39.529428 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dade4233-7aeb-4214-af56-b2b396f27b8a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.539816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dade4233-7aeb-4214-af56-b2b396f27b8a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.545183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7512f6f-d5e1-4437-a812-275c7e0bbf9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.545791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7512f6f-d5e1-4437-a812-275c7e0bbf9c admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.546075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7512f6f-d5e1-4437-a812-275c7e0bbf9c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.546348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7512f6f-d5e1-4437-a812-275c7e0bbf9c admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.547039 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7512f6f-d5e1-4437-a812-275c7e0bbf9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.548640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7512f6f-d5e1-4437-a812-275c7e0bbf9c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.559818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 931/1862] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:39.563920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d133b08c-ab1b-4ad0-96fb-5b0d842c999d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.574100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d133b08c-ab1b-4ad0-96fb-5b0d842c999d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.579253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d525267e-a330-4dde-b95d-0de69e60df77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.579903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d525267e-a330-4dde-b95d-0de69e60df77 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.580184 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d525267e-a330-4dde-b95d-0de69e60df77 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.580459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d525267e-a330-4dde-b95d-0de69e60df77 admin admin] PATH_INFO: `/v3/projects/1f9434f47212421cbde4ca66664895ca/users/904a44147b4b400d89251704ccafab9c/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.595958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d525267e-a330-4dde-b95d-0de69e60df77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f9434f47212421cbde4ca66664895ca, user_id=904a44147b4b400d89251704ccafab9c, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.598606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d525267e-a330-4dde-b95d-0de69e60df77 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.611218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 932/1863] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:39 2022] PUT /identity/v3/projects/1f9434f47212421cbde4ca66664895ca/users/904a44147b4b400d89251704ccafab9c/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:39.621500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142cf961-ff79-4e10-beef-e52b2d8fc0bf None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.621789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142cf961-ff79-4e10-beef-e52b2d8fc0bf None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.622065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142cf961-ff79-4e10-beef-e52b2d8fc0bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.650199 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-142cf961-ff79-4e10-beef-e52b2d8fc0bf None None] MFA Rules not processed for user `904a44147b4b400d89251704ccafab9c`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:39.651449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-142cf961-ff79-4e10-beef-e52b2d8fc0bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:39.709374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:39.716677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 932/1864] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:39.721314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d525267e-a330-4dde-b95d-0de69e60df77 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.781785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d525267e-a330-4dde-b95d-0de69e60df77 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.792534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b2f2db-7384-4c79-a079-3773b7fed935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.793162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b2f2db-7384-4c79-a079-3773b7fed935 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.793441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b2f2db-7384-4c79-a079-3773b7fed935 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.793711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b2f2db-7384-4c79-a079-3773b7fed935 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.794366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b2f2db-7384-4c79-a079-3773b7fed935 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.796321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b2f2db-7384-4c79-a079-3773b7fed935 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.824806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 933/1865] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:39 2022] POST /identity/v3/projects => generated 383 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:39.829470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142cf961-ff79-4e10-beef-e52b2d8fc0bf None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.840372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142cf961-ff79-4e10-beef-e52b2d8fc0bf None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.897025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195b891d-7fd4-420a-9ab6-4b573239584e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.897693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195b891d-7fd4-420a-9ab6-4b573239584e admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.897976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195b891d-7fd4-420a-9ab6-4b573239584e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.898251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195b891d-7fd4-420a-9ab6-4b573239584e admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.898928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195b891d-7fd4-420a-9ab6-4b573239584e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.900899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195b891d-7fd4-420a-9ab6-4b573239584e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.921294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 933/1866] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:39 2022] POST /identity/v3/users => generated 298 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:39.925568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b2f2db-7384-4c79-a079-3773b7fed935 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.936322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b2f2db-7384-4c79-a079-3773b7fed935 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.940944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a263948-0c2f-4af3-9e6d-4fac14f82c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.941542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a263948-0c2f-4af3-9e6d-4fac14f82c0b admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.941823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a263948-0c2f-4af3-9e6d-4fac14f82c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.942100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a263948-0c2f-4af3-9e6d-4fac14f82c0b admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:39.942803 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a263948-0c2f-4af3-9e6d-4fac14f82c0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:39.944403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a263948-0c2f-4af3-9e6d-4fac14f82c0b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:39.956054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 934/1867] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:39.959932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195b891d-7fd4-420a-9ab6-4b573239584e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:39.970604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195b891d-7fd4-420a-9ab6-4b573239584e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:39.986544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e94e9-9f3e-411f-b723-8ab5d0a1bb1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:39.987372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e94e9-9f3e-411f-b723-8ab5d0a1bb1a admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:39.987678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e94e9-9f3e-411f-b723-8ab5d0a1bb1a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:39.987945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278e94e9-9f3e-411f-b723-8ab5d0a1bb1a admin admin] PATH_INFO: `/v3/projects/98aa51eca56642678e3033c2293ac1d1/users/98adfeacae964872b4cf423158016da7/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:40.038501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278e94e9-9f3e-411f-b723-8ab5d0a1bb1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=98aa51eca56642678e3033c2293ac1d1, user_id=98adfeacae964872b4cf423158016da7, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:40.044570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278e94e9-9f3e-411f-b723-8ab5d0a1bb1a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:40.065510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 934/1868] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:39 2022] PUT /identity/v3/projects/98aa51eca56642678e3033c2293ac1d1/users/98adfeacae964872b4cf423158016da7/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:40.067804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a263948-0c2f-4af3-9e6d-4fac14f82c0b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:40.092452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a263948-0c2f-4af3-9e6d-4fac14f82c0b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:40.182686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:40.202551 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca26567-e36b-40d2-975f-1b92cc27c723 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:40.203430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca26567-e36b-40d2-975f-1b92cc27c723 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:40.203717 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca26567-e36b-40d2-975f-1b92cc27c723 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:40.204571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca26567-e36b-40d2-975f-1b92cc27c723 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:40.205016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca26567-e36b-40d2-975f-1b92cc27c723 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:40.207195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca26567-e36b-40d2-975f-1b92cc27c723 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:40.220314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 935/1869] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:40 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:40.224461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e94e9-9f3e-411f-b723-8ab5d0a1bb1a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:40.235843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278e94e9-9f3e-411f-b723-8ab5d0a1bb1a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:40.240769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:40.241518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:40.241723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:40.241999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] PATH_INFO: `/v3/projects/98aa51eca56642678e3033c2293ac1d1/users/98adfeacae964872b4cf423158016da7/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:40.244045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98aa51eca56642678e3033c2293ac1d1, user_id=98adfeacae964872b4cf423158016da7, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:40.246696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:40.263736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 935/1870] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:40 2022] PUT /identity/v3/projects/98aa51eca56642678e3033c2293ac1d1/users/98adfeacae964872b4cf423158016da7/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:40.275025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280f2bd-f316-4ef5-8968-2422e4ce3b86 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:40.275429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280f2bd-f316-4ef5-8968-2422e4ce3b86 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:40.276078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280f2bd-f316-4ef5-8968-2422e4ce3b86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:40.323906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-6280f2bd-f316-4ef5-8968-2422e4ce3b86 None None] MFA Rules not processed for user `98adfeacae964872b4cf423158016da7`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:40.325343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6280f2bd-f316-4ef5-8968-2422e4ce3b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:40.445466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 936/1871] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:40 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:40.456027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:40.458133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:40.549626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4944fafb-730c-40a0-b0f7-0cc91cf46b73 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:40.553475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2027745d-1496-4c9c-b77a-7d0b45b4b377 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:40.554102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2027745d-1496-4c9c-b77a-7d0b45b4b377 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:40.554379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2027745d-1496-4c9c-b77a-7d0b45b4b377 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:40.554810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2027745d-1496-4c9c-b77a-7d0b45b4b377 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:40.650599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2027745d-1496-4c9c-b77a-7d0b45b4b377 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:40.652148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2027745d-1496-4c9c-b77a-7d0b45b4b377 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:40.663309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 936/1872] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:40 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:49.107492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280f2bd-f316-4ef5-8968-2422e4ce3b86 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:49.109126 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6280f2bd-f316-4ef5-8968-2422e4ce3b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:49.186835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280f2bd-f316-4ef5-8968-2422e4ce3b86 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:49.191500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:49.192142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:49.192429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:49.192706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] PATH_INFO: `/v3/users/4a2356f71a58407e95d4495fb7068c27` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:49.193854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a2356f71a58407e95d4495fb7068c27)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:49.196343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:49.255297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a2356f71a58407e95d4495fb7068c27'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:49.263337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a2356f71a58407e95d4495fb7068c27'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:49.264118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a2356f71a58407e95d4495fb7068c27'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:49.264892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a2356f71a58407e95d4495fb7068c27'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:49.275378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a2356f71a58407e95d4495fb7068c27'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:49.286885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:49.303754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 937/1873] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:49 2022] DELETE /identity/v3/users/4a2356f71a58407e95d4495fb7068c27 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:49.309096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d633c886-cc59-4233-8047-b2219f983c68 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:49.311206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d633c886-cc59-4233-8047-b2219f983c68 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:49.311385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d633c886-cc59-4233-8047-b2219f983c68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:49.325454 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:49.363245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d633c886-cc59-4233-8047-b2219f983c68 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:49.366013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d633c886-cc59-4233-8047-b2219f983c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:49.442478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 937/1874] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:49.453207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:49.454901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:49.603227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adda7289-f353-4223-aaec-3ef0127ac0a8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:49.611240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7fe0c6-51e8-4fd0-97a2-5cfd282b6b9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:49.612446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7fe0c6-51e8-4fd0-97a2-5cfd282b6b9c service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:49.613143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7fe0c6-51e8-4fd0-97a2-5cfd282b6b9c service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:49.613677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7fe0c6-51e8-4fd0-97a2-5cfd282b6b9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:49.631037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7fe0c6-51e8-4fd0-97a2-5cfd282b6b9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:49.632633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7fe0c6-51e8-4fd0-97a2-5cfd282b6b9c service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:49.650918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 938/1875] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:50.008159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d633c886-cc59-4233-8047-b2219f983c68 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:50.010030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d633c886-cc59-4233-8047-b2219f983c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:50.030152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d633c886-cc59-4233-8047-b2219f983c68 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:50.038266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:50.039174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:50.039626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:50.040085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] PATH_INFO: `/v3/projects/2970058e05374180a32e27f8496bc290` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:50.042604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2970058e05374180a32e27f8496bc290)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:50.045079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:50.118512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2970058e05374180a32e27f8496bc290', 'request_id': 'req-27d4aaf7-3a26-4939-b15a-2f6d95126001'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.126072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2970058e05374180a32e27f8496bc290', 'request_id': 'req-27d4aaf7-3a26-4939-b15a-2f6d95126001'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.135636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2970058e05374180a32e27f8496bc290', 'request_id': 'req-27d4aaf7-3a26-4939-b15a-2f6d95126001'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.150565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] The token cache is being invalidate because project 2970058e05374180a32e27f8496bc290 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:50.151090 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.151925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.154318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 938/1876] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:50 2022] DELETE /identity/v3/projects/2970058e05374180a32e27f8496bc290 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:50.158634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7fe0c6-51e8-4fd0-97a2-5cfd282b6b9c service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:50.185081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7fe0c6-51e8-4fd0-97a2-5cfd282b6b9c service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:50.283823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:50.284788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:50.285095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:50.285395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] PATH_INFO: `/v3/users/560baf4e37034dbbbfa8421074f8c8b2` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:50.287042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=560baf4e37034dbbbfa8421074f8c8b2)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:50.290801 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:50.354298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '560baf4e37034dbbbfa8421074f8c8b2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.376692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '560baf4e37034dbbbfa8421074f8c8b2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.377881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '560baf4e37034dbbbfa8421074f8c8b2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.378917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '560baf4e37034dbbbfa8421074f8c8b2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.397628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '560baf4e37034dbbbfa8421074f8c8b2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:50.427934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 939/1877] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:50 2022] DELETE /identity/v3/users/560baf4e37034dbbbfa8421074f8c8b2 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:50.974142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:50.977374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:51.024368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d4aaf7-3a26-4939-b15a-2f6d95126001 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.177003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.177712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.178141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.178422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] PATH_INFO: `/v3/projects/64be32f0d7d4445aa5640f8ff1d9886f` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.180466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64be32f0d7d4445aa5640f8ff1d9886f)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.182393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.226530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:51.229586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:51.239060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:51.264687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:51.286639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64be32f0d7d4445aa5640f8ff1d9886f', 'request_id': 'req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.295295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64be32f0d7d4445aa5640f8ff1d9886f', 'request_id': 'req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.308162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64be32f0d7d4445aa5640f8ff1d9886f', 'request_id': 'req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.327792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] The token cache is being invalidate because project 64be32f0d7d4445aa5640f8ff1d9886f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:51.328267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.329394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-221daaec-4ab0-4aa2-b1b3-fcbc2d521a22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.331833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 939/1878] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:50 2022] DELETE /identity/v3/projects/64be32f0d7d4445aa5640f8ff1d9886f => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:51.340504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab96ec-cd91-4d71-8917-4eba4aa27873 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.340767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab96ec-cd91-4d71-8917-4eba4aa27873 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.341050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab96ec-cd91-4d71-8917-4eba4aa27873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.366370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-68ab96ec-cd91-4d71-8917-4eba4aa27873 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:51.366370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-68ab96ec-cd91-4d71-8917-4eba4aa27873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:51.374519 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c6993f-f252-4926-aa40-56d37f93a9ba admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.459318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.460574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.461245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.462266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.467361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:51.477197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 940/1879] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:51.487301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa661e2-2557-4a70-8cb8-adf32eb826bd None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.487595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa661e2-2557-4a70-8cb8-adf32eb826bd None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.488036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa661e2-2557-4a70-8cb8-adf32eb826bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.523851 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1fa661e2-2557-4a70-8cb8-adf32eb826bd None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:51.525075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1fa661e2-2557-4a70-8cb8-adf32eb826bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:51.541506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 941/1880] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:51.546695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa661e2-2557-4a70-8cb8-adf32eb826bd None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:51.558135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa661e2-2557-4a70-8cb8-adf32eb826bd None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.563882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.564377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.564749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.565011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.566087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:51.566397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.568399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.588018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 942/1881] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:51.589009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:51.589690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.591867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.605857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 940/1882] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:51 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:51.608071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acdbbb6-a4b4-4748-baf6-f7bdf7e01f8d admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.613244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.614361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.614361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.614844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.615653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.618099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.653869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 943/1883] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:51 2022] POST /identity/v3/projects => generated 385 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:51.658185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:51.669958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f116e16-1efc-44c5-ae80-1e1c2fca91e9 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.759343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ca8b27-3951-48d7-8f98-6566378ac168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.760160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ca8b27-3951-48d7-8f98-6566378ac168 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.760638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ca8b27-3951-48d7-8f98-6566378ac168 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.760955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ca8b27-3951-48d7-8f98-6566378ac168 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.761600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ca8b27-3951-48d7-8f98-6566378ac168 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.764795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ca8b27-3951-48d7-8f98-6566378ac168 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.771472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:51.774299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:51.787898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 941/1884] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:51 2022] POST /identity/v3/users => generated 293 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:51.791753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ca8b27-3951-48d7-8f98-6566378ac168 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:51.797410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022ccfa2-1b5f-4e4b-8b6f-0def2d759161 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.804608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ca8b27-3951-48d7-8f98-6566378ac168 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.805045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.806127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.806644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.807160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] PATH_INFO: `/v3/users/b34db43f86264c448bf98693bfed7204` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.808780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b34db43f86264c448bf98693bfed7204)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.810162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6f170-b990-49ad-8710-cfefadbe7666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.810928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6f170-b990-49ad-8710-cfefadbe7666 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.811206 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6f170-b990-49ad-8710-cfefadbe7666 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.811517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6f170-b990-49ad-8710-cfefadbe7666 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.812203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f6f170-b990-49ad-8710-cfefadbe7666 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.813119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.813883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f6f170-b990-49ad-8710-cfefadbe7666 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.815537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:51.828086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 942/1885] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:51.831829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6f170-b990-49ad-8710-cfefadbe7666 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:51.844706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6f170-b990-49ad-8710-cfefadbe7666 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.851624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b432407a-c17e-4810-aba8-c7f98cfe97b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.852384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b432407a-c17e-4810-aba8-c7f98cfe97b6 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.852667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b432407a-c17e-4810-aba8-c7f98cfe97b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.853454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b432407a-c17e-4810-aba8-c7f98cfe97b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.853736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b432407a-c17e-4810-aba8-c7f98cfe97b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.855466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b432407a-c17e-4810-aba8-c7f98cfe97b6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.868986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 943/1886] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:51 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:51.873384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b432407a-c17e-4810-aba8-c7f98cfe97b6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:51.874360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b34db43f86264c448bf98693bfed7204'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.875237 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b34db43f86264c448bf98693bfed7204'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.882828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b34db43f86264c448bf98693bfed7204'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.885075 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b432407a-c17e-4810-aba8-c7f98cfe97b6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:51.891748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d124fe2-70b5-4642-a917-e018a461d1e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:51.892561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d124fe2-70b5-4642-a917-e018a461d1e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.892909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d124fe2-70b5-4642-a917-e018a461d1e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.893154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b34db43f86264c448bf98693bfed7204'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.893360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d124fe2-70b5-4642-a917-e018a461d1e9 admin admin] PATH_INFO: `/v3/projects/381b79afba9646468f0e1bec75948423/users/a325e1f4371d433d8f579453296362e5/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.902122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:51.909173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-096082af-3fef-4857-9865-3fb07d8230a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b34db43f86264c448bf98693bfed7204'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:51.909508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d124fe2-70b5-4642-a917-e018a461d1e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=381b79afba9646468f0e1bec75948423, user_id=a325e1f4371d433d8f579453296362e5, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:51.912367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d124fe2-70b5-4642-a917-e018a461d1e9 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:51.920577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 944/1887] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:51 2022] DELETE /identity/v3/users/b34db43f86264c448bf98693bfed7204 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:51.927240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6119c087-0b1e-41c2-868f-23e1bbed03af None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.927397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6119c087-0b1e-41c2-868f-23e1bbed03af None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.927729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6119c087-0b1e-41c2-868f-23e1bbed03af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.937778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 944/1888] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:51 2022] PUT /identity/v3/projects/381b79afba9646468f0e1bec75948423/users/a325e1f4371d433d8f579453296362e5/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:51.949441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0756e-75d8-400b-b898-ddbc4b75970e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:51.949721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0756e-75d8-400b-b898-ddbc4b75970e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:51.950571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0756e-75d8-400b-b898-ddbc4b75970e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:51.961000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-6119c087-0b1e-41c2-868f-23e1bbed03af None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:51.962243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6119c087-0b1e-41c2-868f-23e1bbed03af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:51.986903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-76e0756e-75d8-400b-b898-ddbc4b75970e None None] MFA Rules not processed for user `a325e1f4371d433d8f579453296362e5`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:51.988238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-76e0756e-75d8-400b-b898-ddbc4b75970e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:52.034785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 945/1889] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:52.044135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6119c087-0b1e-41c2-868f-23e1bbed03af None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.045582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6119c087-0b1e-41c2-868f-23e1bbed03af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:52.088244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 945/1890] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:52.092836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e0756e-75d8-400b-b898-ddbc4b75970e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.095570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-76e0756e-75d8-400b-b898-ddbc4b75970e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:52.132420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e0756e-75d8-400b-b898-ddbc4b75970e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:52.139467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b32d2dd-7439-491c-a3ab-dca69088c230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:52.140172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b32d2dd-7439-491c-a3ab-dca69088c230 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.140554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b32d2dd-7439-491c-a3ab-dca69088c230 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.141767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b32d2dd-7439-491c-a3ab-dca69088c230 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.143030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b32d2dd-7439-491c-a3ab-dca69088c230 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:52.145922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b32d2dd-7439-491c-a3ab-dca69088c230 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:52.160111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6119c087-0b1e-41c2-868f-23e1bbed03af None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:52.172388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d769a5b-7460-4fd5-adb8-93b4896576aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:52.173272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d769a5b-7460-4fd5-adb8-93b4896576aa service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.173790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d769a5b-7460-4fd5-adb8-93b4896576aa service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.174087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d769a5b-7460-4fd5-adb8-93b4896576aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.185970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d769a5b-7460-4fd5-adb8-93b4896576aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:52.187942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d769a5b-7460-4fd5-adb8-93b4896576aa service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:52.194680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 946/1891] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:52 2022] POST /identity/v3/projects => generated 387 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:52.199190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b32d2dd-7439-491c-a3ab-dca69088c230 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.202461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 946/1892] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:41:52.302671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b32d2dd-7439-491c-a3ab-dca69088c230 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:52.308782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f164decb-91e7-4ea5-a952-d3adc299407f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:52.310194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164decb-91e7-4ea5-a952-d3adc299407f admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.310854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164decb-91e7-4ea5-a952-d3adc299407f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.310945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164decb-91e7-4ea5-a952-d3adc299407f admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.311920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f164decb-91e7-4ea5-a952-d3adc299407f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:52.314073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f164decb-91e7-4ea5-a952-d3adc299407f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:52.347537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 947/1893] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:52 2022] POST /identity/v3/users => generated 300 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:52.351143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d769a5b-7460-4fd5-adb8-93b4896576aa service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.371576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d769a5b-7460-4fd5-adb8-93b4896576aa service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:52.381912 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7675f-9eca-47e3-b8cc-57cc52f0e0be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:52.382531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7675f-9eca-47e3-b8cc-57cc52f0e0be admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.383289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7675f-9eca-47e3-b8cc-57cc52f0e0be admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.383693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7675f-9eca-47e3-b8cc-57cc52f0e0be admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.384864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7675f-9eca-47e3-b8cc-57cc52f0e0be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:52.386871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7675f-9eca-47e3-b8cc-57cc52f0e0be admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:52.407319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 947/1894] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:52 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:52.412877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f164decb-91e7-4ea5-a952-d3adc299407f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.431004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f164decb-91e7-4ea5-a952-d3adc299407f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:52.437783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:52.439021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.439697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.440575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] PATH_INFO: `/v3/projects/592a636d6cc8463cb8bb6256505c3fe7/users/51d1811fff0f4c03b842f462477d8201/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.481680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] RBAC: Authorizing `identity:create_grant(project_id=592a636d6cc8463cb8bb6256505c3fe7, user_id=51d1811fff0f4c03b842f462477d8201, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:52.481680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:52.481680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:52.508841 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 948/1895] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:52 2022] PUT /identity/v3/projects/592a636d6cc8463cb8bb6256505c3fe7/users/51d1811fff0f4c03b842f462477d8201/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:52.513690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7675f-9eca-47e3-b8cc-57cc52f0e0be admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.543491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7675f-9eca-47e3-b8cc-57cc52f0e0be admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:52.630775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.632817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:52.740116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7799aff-e89f-44d2-aca5-7bc49ed35394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:52.741123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7799aff-e89f-44d2-aca5-7bc49ed35394 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.741833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7799aff-e89f-44d2-aca5-7bc49ed35394 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.742675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7799aff-e89f-44d2-aca5-7bc49ed35394 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.744226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7799aff-e89f-44d2-aca5-7bc49ed35394 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:52.746218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7799aff-e89f-44d2-aca5-7bc49ed35394 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:52.777108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 948/1896] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:52 2022] GET /identity/v3/roles => generated 1420 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:52.778300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7799aff-e89f-44d2-aca5-7bc49ed35394 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.778931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9ca083-dd49-444f-bf31-1ff8c82d875f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:52.790368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:52.791283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.791692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.792000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] PATH_INFO: `/v3/projects/31484448939c42e39d5977a1c8ccae52` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.793413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=31484448939c42e39d5977a1c8ccae52)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:52.793575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7799aff-e89f-44d2-aca5-7bc49ed35394 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:52.804264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:52.806195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e641c4e2-e37e-4981-9b47-cb585f478d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:52.807009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641c4e2-e37e-4981-9b47-cb585f478d6e admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.807296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641c4e2-e37e-4981-9b47-cb585f478d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.808080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641c4e2-e37e-4981-9b47-cb585f478d6e admin admin] PATH_INFO: `/v3/projects/592a636d6cc8463cb8bb6256505c3fe7/users/51d1811fff0f4c03b842f462477d8201/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.812496 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e641c4e2-e37e-4981-9b47-cb585f478d6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=592a636d6cc8463cb8bb6256505c3fe7, user_id=51d1811fff0f4c03b842f462477d8201, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:52.819881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e641c4e2-e37e-4981-9b47-cb585f478d6e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:52.841614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 949/1897] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:52 2022] PUT /identity/v3/projects/592a636d6cc8463cb8bb6256505c3fe7/users/51d1811fff0f4c03b842f462477d8201/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:52.852278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbd9748-e89c-4406-b31e-76e7b9051ab8 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:52.852988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbd9748-e89c-4406-b31e-76e7b9051ab8 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:52.853169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbd9748-e89c-4406-b31e-76e7b9051ab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:52.865323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:52.893285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31484448939c42e39d5977a1c8ccae52', 'request_id': 'req-9d249a1a-700e-4b90-b896-00da96b8c8d1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:52.907825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7bbd9748-e89c-4406-b31e-76e7b9051ab8 None None] MFA Rules not processed for user `51d1811fff0f4c03b842f462477d8201`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:52.909280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7bbd9748-e89c-4406-b31e-76e7b9051ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:52.912232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31484448939c42e39d5977a1c8ccae52', 'request_id': 'req-9d249a1a-700e-4b90-b896-00da96b8c8d1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:52.923776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31484448939c42e39d5977a1c8ccae52', 'request_id': 'req-9d249a1a-700e-4b90-b896-00da96b8c8d1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:52.956899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] The token cache is being invalidate because project 31484448939c42e39d5977a1c8ccae52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:52.957745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:52.959891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:52.965144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 949/1898] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:52 2022] DELETE /identity/v3/projects/31484448939c42e39d5977a1c8ccae52 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:52.970548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:52.973681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:53.030830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 950/1899] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:52 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:53.042418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbd9748-e89c-4406-b31e-76e7b9051ab8 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:53.044478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7bbd9748-e89c-4406-b31e-76e7b9051ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:53.180643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d249a1a-700e-4b90-b896-00da96b8c8d1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:53.184067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbd9748-e89c-4406-b31e-76e7b9051ab8 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:53.188299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:53.190294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:53.190584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:53.190897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] PATH_INFO: `/v3/users/277e24c143ca427eaae9c9f9caf79c50` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:53.192223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:53.194553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:53.195211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:53.195211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:53.196952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=277e24c143ca427eaae9c9f9caf79c50)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:53.201493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:53.220916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:53.225504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:53.242901 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 951/1900] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:53 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:53.268021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:INSERT=2,keystone:SELECT=87,keystone:DELETE=11 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:53.299635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '277e24c143ca427eaae9c9f9caf79c50'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:53.320094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '277e24c143ca427eaae9c9f9caf79c50'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:53.320868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '277e24c143ca427eaae9c9f9caf79c50'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:53.321700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '277e24c143ca427eaae9c9f9caf79c50'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:53.339215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '277e24c143ca427eaae9c9f9caf79c50'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:53.380284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 950/1901] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:52 2022] DELETE /identity/v3/users/277e24c143ca427eaae9c9f9caf79c50 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:53.876117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:53.877620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:53.894082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8807aeeb-791f-407c-b98e-bfdd2d12b7ca service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:53.999690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:54.000967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.001536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.002761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] PATH_INFO: `/v3/users/7e9057f1e5704e44b8951d26295d2ca3` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.005361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e9057f1e5704e44b8951d26295d2ca3)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:54.009292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:54.010339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:54.014660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:54.076254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3139d4-dca5-4f17-8b47-3d0d3ea535c9 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:54.092535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:54.095617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.096131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.096832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] PATH_INFO: `/v3/projects/9152791b9472426d91b410ab314aeb64` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.098251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9152791b9472426d91b410ab314aeb64)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:54.100793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:54.135348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:54.160330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e9057f1e5704e44b8951d26295d2ca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.162232 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e9057f1e5704e44b8951d26295d2ca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.180426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e9057f1e5704e44b8951d26295d2ca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.195989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e9057f1e5704e44b8951d26295d2ca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.213484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8599cb97-8e45-4d74-84fe-2a3da7d3397f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e9057f1e5704e44b8951d26295d2ca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.213942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9152791b9472426d91b410ab314aeb64', 'request_id': 'req-ead5037e-fe88-45ab-ae06-1daf3fb75263'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.226282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 952/1902] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:53 2022] DELETE /identity/v3/users/7e9057f1e5704e44b8951d26295d2ca3 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:54.231768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee33ff8-1d1b-4664-8896-533ec167b222 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.233732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee33ff8-1d1b-4664-8896-533ec167b222 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.234520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee33ff8-1d1b-4664-8896-533ec167b222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.245636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9152791b9472426d91b410ab314aeb64', 'request_id': 'req-ead5037e-fe88-45ab-ae06-1daf3fb75263'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.260082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9152791b9472426d91b410ab314aeb64', 'request_id': 'req-ead5037e-fe88-45ab-ae06-1daf3fb75263'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.262568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5ee33ff8-1d1b-4664-8896-533ec167b222 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:54.264149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5ee33ff8-1d1b-4664-8896-533ec167b222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:54.291476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] The token cache is being invalidate because project 9152791b9472426d91b410ab314aeb64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:54.291476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.294847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ead5037e-fe88-45ab-ae06-1daf3fb75263 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.302068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 951/1903] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:54 2022] DELETE /identity/v3/projects/9152791b9472426d91b410ab314aeb64 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:54.304139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bafe035-80de-4d7d-a827-7c94edad0178 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.304838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bafe035-80de-4d7d-a827-7c94edad0178 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.305205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bafe035-80de-4d7d-a827-7c94edad0178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.330561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7bafe035-80de-4d7d-a827-7c94edad0178 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:54.332007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7bafe035-80de-4d7d-a827-7c94edad0178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:54.400622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 953/1904] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:54.423732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee33ff8-1d1b-4664-8896-533ec167b222 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:54.428431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5ee33ff8-1d1b-4664-8896-533ec167b222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:54.437588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 952/1905] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:54.447212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66438092-04ff-4c1b-8f68-5b2e9c43c107 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.448401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66438092-04ff-4c1b-8f68-5b2e9c43c107 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.448582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66438092-04ff-4c1b-8f68-5b2e9c43c107 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.482005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-66438092-04ff-4c1b-8f68-5b2e9c43c107 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:54.484420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-66438092-04ff-4c1b-8f68-5b2e9c43c107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:54.511356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 953/1906] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:54.515889 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66438092-04ff-4c1b-8f68-5b2e9c43c107 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:54.550153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66438092-04ff-4c1b-8f68-5b2e9c43c107 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:54.571428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:54.572234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.573275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.574760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.575724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:54.578403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:54.582858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:54.587260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee33ff8-1d1b-4664-8896-533ec167b222 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:54.596443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb7da53-64f3-4699-961a-82e61bea3013 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:54.597156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb7da53-64f3-4699-961a-82e61bea3013 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.597645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb7da53-64f3-4699-961a-82e61bea3013 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.597991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb7da53-64f3-4699-961a-82e61bea3013 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.610786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 954/1907] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:54.615850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:54.618970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:54.620067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb7da53-64f3-4699-961a-82e61bea3013 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:54.621711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb7da53-64f3-4699-961a-82e61bea3013 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:54.634284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 954/1908] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:54.635690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6546149-a0aa-45ff-965b-85b13691699f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:54.648730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:54.649632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb7da53-64f3-4699-961a-82e61bea3013 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:54.649903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.654589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.654952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] PATH_INFO: `/v3/users/a325e1f4371d433d8f579453296362e5` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.656525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a325e1f4371d433d8f579453296362e5)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:54.659142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:54.661566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:54.669010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb7da53-64f3-4699-961a-82e61bea3013 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:54.681350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d2507b-cb33-42a9-a5ed-720742baf95f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:54.681975 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d2507b-cb33-42a9-a5ed-720742baf95f admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.682256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d2507b-cb33-42a9-a5ed-720742baf95f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.682604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d2507b-cb33-42a9-a5ed-720742baf95f admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.683196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d2507b-cb33-42a9-a5ed-720742baf95f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:54.685273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d2507b-cb33-42a9-a5ed-720742baf95f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:54.747830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a325e1f4371d433d8f579453296362e5'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.750065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 955/1909] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:54 2022] POST /identity/v3/projects => generated 357 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:54.754392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d2507b-cb33-42a9-a5ed-720742baf95f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:54.778934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a325e1f4371d433d8f579453296362e5'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.780405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a325e1f4371d433d8f579453296362e5'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.782854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a325e1f4371d433d8f579453296362e5'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.802137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a943b945-e500-482f-a0bf-0b7aabe52430 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a325e1f4371d433d8f579453296362e5'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:54.829557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:54.852774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 955/1910] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:54 2022] DELETE /identity/v3/users/a325e1f4371d433d8f579453296362e5 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:41:54.859297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662ae1f-4c8f-419e-b694-95b3c00646da None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:54.860109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662ae1f-4c8f-419e-b694-95b3c00646da None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:54.860605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662ae1f-4c8f-419e-b694-95b3c00646da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:54.879292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d2507b-cb33-42a9-a5ed-720742baf95f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:54.901522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c662ae1f-4c8f-419e-b694-95b3c00646da None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:54.903078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c662ae1f-4c8f-419e-b694-95b3c00646da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.010953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:55.011579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.011861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.012162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.013341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:55.015363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:55.031645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 956/1911] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:55.044697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 956/1912] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:54 2022] POST /identity/v3/users => generated 279 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:55.079139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c662ae1f-4c8f-419e-b694-95b3c00646da None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.079980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.088695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c662ae1f-4c8f-419e-b694-95b3c00646da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.089071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.117672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c662ae1f-4c8f-419e-b694-95b3c00646da None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:55.131432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:55.132482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.132952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.133449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.134305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:55.136504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:55.185211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 957/1913] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:55 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:55.185211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.189179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.239349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02656ed-c791-4a02-a7d7-23d0bb63ca07 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:55.245882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:55.247495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.247645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.248077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] PATH_INFO: `/v3/projects/5c029a4bb1ce4d48abae261c46a905f5` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.248239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf7fe84-854c-4d50-95fa-2d51005e99f8 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:55.249157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c029a4bb1ce4d48abae261c46a905f5)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:55.251384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:55.259467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a932ffed-3ec8-4671-9d43-334a2dd9d164 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:55.260134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a932ffed-3ec8-4671-9d43-334a2dd9d164 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.260458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a932ffed-3ec8-4671-9d43-334a2dd9d164 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.260698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a932ffed-3ec8-4671-9d43-334a2dd9d164 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.277520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a932ffed-3ec8-4671-9d43-334a2dd9d164 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:55.279197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a932ffed-3ec8-4671-9d43-334a2dd9d164 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:55.291831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 957/1914] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:55.296145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a932ffed-3ec8-4671-9d43-334a2dd9d164 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.313297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a932ffed-3ec8-4671-9d43-334a2dd9d164 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:55.320422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c029a4bb1ce4d48abae261c46a905f5', 'request_id': 'req-9ef14c9d-5239-457b-9371-41fd80e28dc5'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.331575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c029a4bb1ce4d48abae261c46a905f5', 'request_id': 'req-9ef14c9d-5239-457b-9371-41fd80e28dc5'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.345835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c029a4bb1ce4d48abae261c46a905f5', 'request_id': 'req-9ef14c9d-5239-457b-9371-41fd80e28dc5'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.349295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:55.378034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] The token cache is being invalidate because project 5c029a4bb1ce4d48abae261c46a905f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:55.378034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.380081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.388816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 958/1915] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:55 2022] DELETE /identity/v3/projects/5c029a4bb1ce4d48abae261c46a905f5 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:55.390257 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.393188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.423535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:55.424573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.425073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.425568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.426772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:55.431105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:55.448495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 958/1916] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:55 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:55.455333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.457681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.473411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e393815-63c4-4aa4-9244-3f5aebf8cc01 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:55.483146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c645c4-4183-4c87-a255-e5be747d1af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:55.483885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c645c4-4183-4c87-a255-e5be747d1af5 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.484216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c645c4-4183-4c87-a255-e5be747d1af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.484495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c645c4-4183-4c87-a255-e5be747d1af5 admin admin] PATH_INFO: `/v3/projects/0da1a62f992b4d3f86adaa2d0230d35b/users/80f895cc698644b189c81240b9192823/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.495831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:55.510594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c645c4-4183-4c87-a255-e5be747d1af5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0da1a62f992b4d3f86adaa2d0230d35b, user_id=80f895cc698644b189c81240b9192823, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:55.514476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c645c4-4183-4c87-a255-e5be747d1af5 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:55.538881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 959/1917] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:55 2022] PUT /identity/v3/projects/0da1a62f992b4d3f86adaa2d0230d35b/users/80f895cc698644b189c81240b9192823/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:55.555501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d759d-2360-4e5c-9acd-5b3206315b71 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.555945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d759d-2360-4e5c-9acd-5b3206315b71 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.556224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d759d-2360-4e5c-9acd-5b3206315b71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.579052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef14c9d-5239-457b-9371-41fd80e28dc5 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:55.625341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3f4d759d-2360-4e5c-9acd-5b3206315b71 None None] MFA Rules not processed for user `80f895cc698644b189c81240b9192823`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:55.626791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3f4d759d-2360-4e5c-9acd-5b3206315b71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.695899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:55.699574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.707378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.707759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] PATH_INFO: `/v3/users/66ad8a4b41104aab8bdc818bbff338ca` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.709414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] RBAC: Authorizing `identity:delete_user(user_id=66ad8a4b41104aab8bdc818bbff338ca)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:55.712830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:55.751352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 960/1918] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:55 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:41:55.752450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4d759d-2360-4e5c-9acd-5b3206315b71 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.755257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3f4d759d-2360-4e5c-9acd-5b3206315b71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.794841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4d759d-2360-4e5c-9acd-5b3206315b71 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:55.810864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:55.811530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:55.811803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:55.812074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] PATH_INFO: `/v3/projects/381b79afba9646468f0e1bec75948423` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:55.814941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] RBAC: Authorizing `identity:delete_project(project_id=381b79afba9646468f0e1bec75948423)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:55.816097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:55.816448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '66ad8a4b41104aab8bdc818bbff338ca'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.831765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '66ad8a4b41104aab8bdc818bbff338ca'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.833922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '66ad8a4b41104aab8bdc818bbff338ca'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.834851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '66ad8a4b41104aab8bdc818bbff338ca'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.849510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '66ad8a4b41104aab8bdc818bbff338ca'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.876137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 959/1919] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:55 2022] DELETE /identity/v3/users/66ad8a4b41104aab8bdc818bbff338ca => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:55.878060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.879955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:55.912425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:INSERT=2,keystone:DELETE=11,keystone:SELECT=87 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:55.925155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '381b79afba9646468f0e1bec75948423', 'request_id': 'req-ef3ba29a-6810-419f-99b4-f40108854367'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.933770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '381b79afba9646468f0e1bec75948423', 'request_id': 'req-ef3ba29a-6810-419f-99b4-f40108854367'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.945606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '381b79afba9646468f0e1bec75948423', 'request_id': 'req-ef3ba29a-6810-419f-99b4-f40108854367'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.961626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] The token cache is being invalidate because project 381b79afba9646468f0e1bec75948423 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:55.961936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.962864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:55.964879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 961/1920] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:55 2022] DELETE /identity/v3/projects/381b79afba9646468f0e1bec75948423 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:55.970033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:55.972426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:56.008221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f4ddec-2a4e-428d-b531-e330f443e778 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:56.058497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:56.066180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3ba29a-6810-419f-99b4-f40108854367 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:56.081156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:56.082070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:56.082547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:56.082962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] PATH_INFO: `/v3/users/51d1811fff0f4c03b842f462477d8201` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:56.085197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51d1811fff0f4c03b842f462477d8201)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:56.088846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:56.159963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:56.160580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:56.160944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:56.161195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:56.161789 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:56.166781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:56.240013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 960/1921] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:55 2022] POST /identity/v3/projects => generated 355 bytes in 363 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:56.244810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:56.255688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51d1811fff0f4c03b842f462477d8201'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.261733 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:56.262673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51d1811fff0f4c03b842f462477d8201'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.285693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51d1811fff0f4c03b842f462477d8201'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.302575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51d1811fff0f4c03b842f462477d8201'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.322831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51d1811fff0f4c03b842f462477d8201'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.332362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 962/1922] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:55 2022] DELETE /identity/v3/users/51d1811fff0f4c03b842f462477d8201 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:56.454594 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4837fc-35c5-4ce2-9a09-22de915b5b46 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:56.548974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:56.550689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:56.603794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:56.614124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:56.615322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:56.615899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:56.616175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:56.616829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:56.620107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:56.658581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 961/1923] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:56 2022] POST /identity/v3/users => generated 284 bytes in 411 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:56.659663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:56.662604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:56.681549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59e2ade-28b0-4eb9-bc8e-8b0e9d16ac02 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:56.687326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0effdc-eedb-4a40-a52d-e5509d9660e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:56.687969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0effdc-eedb-4a40-a52d-e5509d9660e5 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:56.688244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0effdc-eedb-4a40-a52d-e5509d9660e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:56.688527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0effdc-eedb-4a40-a52d-e5509d9660e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:56.689183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0effdc-eedb-4a40-a52d-e5509d9660e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:56.690239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbe4eea-a420-4a20-9d54-7a70663e7816 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:56.690837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0effdc-eedb-4a40-a52d-e5509d9660e5 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:56.706915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 962/1924] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:56 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:56.709390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0effdc-eedb-4a40-a52d-e5509d9660e5 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:56.719766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:56.719766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:56.719766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:56.719766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] PATH_INFO: `/v3/projects/0f8147eaf48b4aa0af74d4925ef26a66` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:56.721987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f8147eaf48b4aa0af74d4925ef26a66)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:56.724220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:56.732386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0effdc-eedb-4a40-a52d-e5509d9660e5 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:56.744271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14734a36-ca0f-48f0-8578-fc3de3151289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:56.744967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14734a36-ca0f-48f0-8578-fc3de3151289 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:56.745251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14734a36-ca0f-48f0-8578-fc3de3151289 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:56.745525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14734a36-ca0f-48f0-8578-fc3de3151289 admin admin] PATH_INFO: `/v3/projects/6fc71ea0219443bca187d837a00e443b/users/bb0944cbbc9f4cbf8a1859e39d858fd2/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:56.795307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14734a36-ca0f-48f0-8578-fc3de3151289 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fc71ea0219443bca187d837a00e443b, user_id=bb0944cbbc9f4cbf8a1859e39d858fd2, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:56.798397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14734a36-ca0f-48f0-8578-fc3de3151289 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:56.820209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f8147eaf48b4aa0af74d4925ef26a66', 'request_id': 'req-099f4001-b94f-431e-846d-4702eaefa98f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.827397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 963/1925] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:56 2022] PUT /identity/v3/projects/6fc71ea0219443bca187d837a00e443b/users/bb0944cbbc9f4cbf8a1859e39d858fd2/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:56.828512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14734a36-ca0f-48f0-8578-fc3de3151289 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:56.852137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f8147eaf48b4aa0af74d4925ef26a66', 'request_id': 'req-099f4001-b94f-431e-846d-4702eaefa98f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.869023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f8147eaf48b4aa0af74d4925ef26a66', 'request_id': 'req-099f4001-b94f-431e-846d-4702eaefa98f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.905435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] The token cache is being invalidate because project 0f8147eaf48b4aa0af74d4925ef26a66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:56.905950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.907044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:56.909734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 963/1926] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:56 2022] DELETE /identity/v3/projects/0f8147eaf48b4aa0af74d4925ef26a66 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:56.911115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:56.913929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:56.930942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:56.950516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14734a36-ca0f-48f0-8578-fc3de3151289 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:56.974468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7eed09-ac21-4c62-a536-97797b958d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:56.978464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7eed09-ac21-4c62-a536-97797b958d78 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:56.982166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7eed09-ac21-4c62-a536-97797b958d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:56.982959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7eed09-ac21-4c62-a536-97797b958d78 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:56.984410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7eed09-ac21-4c62-a536-97797b958d78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:56.987132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7eed09-ac21-4c62-a536-97797b958d78 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.011578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 964/1927] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:56 2022] GET /identity/v3/roles => generated 1420 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:41:57.013283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664b2835-9f73-454b-85a6-c83e602646b5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.013859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664b2835-9f73-454b-85a6-c83e602646b5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.014393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664b2835-9f73-454b-85a6-c83e602646b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.031358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099f4001-b94f-431e-846d-4702eaefa98f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.038947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.039912 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.040431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.040811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] PATH_INFO: `/v3/projects/592a636d6cc8463cb8bb6256505c3fe7` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.042049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-664b2835-9f73-454b-85a6-c83e602646b5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:57.042639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=592a636d6cc8463cb8bb6256505c3fe7)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.043461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-664b2835-9f73-454b-85a6-c83e602646b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.045125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.052536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 965/1928] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:57.053352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664b2835-9f73-454b-85a6-c83e602646b5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.055395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-664b2835-9f73-454b-85a6-c83e602646b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.070818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664b2835-9f73-454b-85a6-c83e602646b5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.084439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b5b7de-a030-4a51-a307-6328ecca2a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.086655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b5b7de-a030-4a51-a307-6328ecca2a10 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.086958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b5b7de-a030-4a51-a307-6328ecca2a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.087227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b5b7de-a030-4a51-a307-6328ecca2a10 admin admin] PATH_INFO: `/v3/projects/6fc71ea0219443bca187d837a00e443b/users/bb0944cbbc9f4cbf8a1859e39d858fd2/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.089816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b5b7de-a030-4a51-a307-6328ecca2a10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fc71ea0219443bca187d837a00e443b, user_id=bb0944cbbc9f4cbf8a1859e39d858fd2, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.093335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b5b7de-a030-4a51-a307-6328ecca2a10 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.109614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 966/1929] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:57 2022] PUT /identity/v3/projects/6fc71ea0219443bca187d837a00e443b/users/bb0944cbbc9f4cbf8a1859e39d858fd2/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:57.117282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c73ed-2b75-4713-bda0-dce592e7a6e9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.117766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c73ed-2b75-4713-bda0-dce592e7a6e9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.118665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c73ed-2b75-4713-bda0-dce592e7a6e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.142054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '592a636d6cc8463cb8bb6256505c3fe7', 'request_id': 'req-7882670e-0bd4-4a9d-b9c6-638df401c0d5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.147958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-de6c73ed-2b75-4713-bda0-dce592e7a6e9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:57.149600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-de6c73ed-2b75-4713-bda0-dce592e7a6e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.150657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '592a636d6cc8463cb8bb6256505c3fe7', 'request_id': 'req-7882670e-0bd4-4a9d-b9c6-638df401c0d5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.160074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '592a636d6cc8463cb8bb6256505c3fe7', 'request_id': 'req-7882670e-0bd4-4a9d-b9c6-638df401c0d5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.175536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] The token cache is being invalidate because project 592a636d6cc8463cb8bb6256505c3fe7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:57.175907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.176721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7882670e-0bd4-4a9d-b9c6-638df401c0d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.178351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 964/1930] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:56 2022] DELETE /identity/v3/projects/592a636d6cc8463cb8bb6256505c3fe7 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:57.182762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a54734-3a30-4af2-a25f-e452bbc88e44 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.183063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a54734-3a30-4af2-a25f-e452bbc88e44 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.183361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a54734-3a30-4af2-a25f-e452bbc88e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.227294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-95a54734-3a30-4af2-a25f-e452bbc88e44 None None] MFA Rules not processed for user `bb0944cbbc9f4cbf8a1859e39d858fd2`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:57.229104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-95a54734-3a30-4af2-a25f-e452bbc88e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.233829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 967/1931] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:57.238415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5d78d-db56-4c73-9e3c-2a81853eb71a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.239413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5d78d-db56-4c73-9e3c-2a81853eb71a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.239814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5d78d-db56-4c73-9e3c-2a81853eb71a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.259967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5cd5d78d-db56-4c73-9e3c-2a81853eb71a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:57.261512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5cd5d78d-db56-4c73-9e3c-2a81853eb71a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.270122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 968/1932] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:57.271053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd5d78d-db56-4c73-9e3c-2a81853eb71a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.284192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd5d78d-db56-4c73-9e3c-2a81853eb71a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.289798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e092db-0dfa-4a55-813d-88defc072a92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.294147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e092db-0dfa-4a55-813d-88defc072a92 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.300948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e092db-0dfa-4a55-813d-88defc072a92 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.301442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e092db-0dfa-4a55-813d-88defc072a92 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.302601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e092db-0dfa-4a55-813d-88defc072a92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:57.303050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e092db-0dfa-4a55-813d-88defc072a92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.305911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e092db-0dfa-4a55-813d-88defc072a92 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.327905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 969/1933] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:57.329234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e116286d-275e-4f82-b3d1-0217817702d8 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.329527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e116286d-275e-4f82-b3d1-0217817702d8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.330373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e116286d-275e-4f82-b3d1-0217817702d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.356767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e116286d-275e-4f82-b3d1-0217817702d8 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:57.358054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e116286d-275e-4f82-b3d1-0217817702d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.368080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 970/1934] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:57.368733 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e116286d-275e-4f82-b3d1-0217817702d8 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.374312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:57.382373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e116286d-275e-4f82-b3d1-0217817702d8 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.388647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.388948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 965/1935] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:57.389633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.389810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.390055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.391419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.391662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a54734-3a30-4af2-a25f-e452bbc88e44 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.403984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.409946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a54734-3a30-4af2-a25f-e452bbc88e44 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.416939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.417788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.418064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.418895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.419575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:41:57.419885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.421603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.438702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 971/1936] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/projects => generated 385 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:57.439666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.442675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.447248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 966/1937] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:41:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:41:57.448131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.466438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5ec056-a813-4370-ba5d-66626caf5829 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.473082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:57.507445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4668c1a6-9310-4be1-8614-83bc5306e4bd admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.513956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.514601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.514931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.515181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.567470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7e1271-ee8c-4c01-ae11-32b7600bf5f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.568875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7e1271-ee8c-4c01-ae11-32b7600bf5f3 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.569338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7e1271-ee8c-4c01-ae11-32b7600bf5f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.569908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7e1271-ee8c-4c01-ae11-32b7600bf5f3 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.571294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7e1271-ee8c-4c01-ae11-32b7600bf5f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.576787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7e1271-ee8c-4c01-ae11-32b7600bf5f3 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.610617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 967/1938] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/users => generated 293 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:57.615718 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7e1271-ee8c-4c01-ae11-32b7600bf5f3 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.620280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.622002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.629934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7e1271-ee8c-4c01-ae11-32b7600bf5f3 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.633405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 972/1939] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:57 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:57.634460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.636098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa60bdd-4008-44d4-9f61-059aec852f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.636805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa60bdd-4008-44d4-9f61-059aec852f0c admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.637593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.637858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa60bdd-4008-44d4-9f61-059aec852f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.638241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa60bdd-4008-44d4-9f61-059aec852f0c admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.638994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa60bdd-4008-44d4-9f61-059aec852f0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.641117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa60bdd-4008-44d4-9f61-059aec852f0c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.656784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92c0292-f7f3-40d5-8d9e-519a7dbba474 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.663327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.666218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.667504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.667865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] PATH_INFO: `/v3/users/904a44147b4b400d89251704ccafab9c` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.669448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=904a44147b4b400d89251704ccafab9c)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.677342 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.688788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 968/1940] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/projects => generated 381 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:57.689907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa60bdd-4008-44d4-9f61-059aec852f0c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.753057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '904a44147b4b400d89251704ccafab9c'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.763704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '904a44147b4b400d89251704ccafab9c'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.764411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '904a44147b4b400d89251704ccafab9c'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.764998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '904a44147b4b400d89251704ccafab9c'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.774856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '904a44147b4b400d89251704ccafab9c'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:57.791064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa60bdd-4008-44d4-9f61-059aec852f0c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.797183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.797954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.798358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.798704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.799420 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.801068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.802769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 973/1941] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:57 2022] DELETE /identity/v3/users/904a44147b4b400d89251704ccafab9c => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:57.804113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.805049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.813100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:57.820334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 969/1942] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:57 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:57.820334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:57.826950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:57.924683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e674cf7-fae9-4cf9-bef8-98bbbd1cac02 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.929947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f588d0f3-e1ca-470d-b889-a372ef6d09a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:57.931157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588d0f3-e1ca-470d-b889-a372ef6d09a0 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.931644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588d0f3-e1ca-470d-b889-a372ef6d09a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.932221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588d0f3-e1ca-470d-b889-a372ef6d09a0 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.933048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f588d0f3-e1ca-470d-b889-a372ef6d09a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:57.935537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f588d0f3-e1ca-470d-b889-a372ef6d09a0 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:57.940193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d6bba2-4c9a-4ca4-91cd-0a41f8633093 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:57.979261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 974/1943] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/users => generated 291 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:57.987617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46fc628-b749-40d4-9cdc-11a3723ee784 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:57.987727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46fc628-b749-40d4-9cdc-11a3723ee784 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:57.987948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46fc628-b749-40d4-9cdc-11a3723ee784 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:57.990956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:58.024171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.026019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.026019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.026337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.029854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:58.046572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d46fc628-b749-40d4-9cdc-11a3723ee784 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:58.049405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d46fc628-b749-40d4-9cdc-11a3723ee784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:58.080452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 975/1944] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:58.086612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46fc628-b749-40d4-9cdc-11a3723ee784 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.094837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d46fc628-b749-40d4-9cdc-11a3723ee784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:58.109545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46fc628-b749-40d4-9cdc-11a3723ee784 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.114664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ce666d-e633-4dbc-8fe4-0e42eb9c4233 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.115478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ce666d-e633-4dbc-8fe4-0e42eb9c4233 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.115770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ce666d-e633-4dbc-8fe4-0e42eb9c4233 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.116046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ce666d-e633-4dbc-8fe4-0e42eb9c4233 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.116698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ce666d-e633-4dbc-8fe4-0e42eb9c4233 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.119161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ce666d-e633-4dbc-8fe4-0e42eb9c4233 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.138044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 976/1945] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:58 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:58.139198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ce666d-e633-4dbc-8fe4-0e42eb9c4233 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.141909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.143727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.150688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ce666d-e633-4dbc-8fe4-0e42eb9c4233 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.156929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 970/1946] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:57 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:58.158569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58932b0d-44c7-454a-8c0d-a015b8d2c215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.159382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58932b0d-44c7-454a-8c0d-a015b8d2c215 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.159812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58932b0d-44c7-454a-8c0d-a015b8d2c215 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.160098 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58932b0d-44c7-454a-8c0d-a015b8d2c215 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.160814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58932b0d-44c7-454a-8c0d-a015b8d2c215 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.162504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58932b0d-44c7-454a-8c0d-a015b8d2c215 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.171552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.174319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:58.176923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 977/1947] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:58 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:58.177794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58932b0d-44c7-454a-8c0d-a015b8d2c215 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.197882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58932b0d-44c7-454a-8c0d-a015b8d2c215 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.211795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eb4092-f1f0-40c6-b73c-856f354f8168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.212476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb4092-f1f0-40c6-b73c-856f354f8168 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.212756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb4092-f1f0-40c6-b73c-856f354f8168 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.213032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb4092-f1f0-40c6-b73c-856f354f8168 admin admin] PATH_INFO: `/v3/projects/fda8b4f99c564368a9a1f08e8789cfe1/users/415b09885347444481719e72ecb1475a/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.233948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23eb4092-f1f0-40c6-b73c-856f354f8168 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fda8b4f99c564368a9a1f08e8789cfe1, user_id=415b09885347444481719e72ecb1475a, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.239095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23eb4092-f1f0-40c6-b73c-856f354f8168 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.260044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 978/1948] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:58 2022] PUT /identity/v3/projects/fda8b4f99c564368a9a1f08e8789cfe1/users/415b09885347444481719e72ecb1475a/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:58.260880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eb4092-f1f0-40c6-b73c-856f354f8168 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.276068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ca8aa-dde7-4205-8440-a14fea38bc82 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.369593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eb4092-f1f0-40c6-b73c-856f354f8168 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.376498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17d10a5-a840-4bf5-bd2d-5f102fb7b29f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.377254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17d10a5-a840-4bf5-bd2d-5f102fb7b29f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.377774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17d10a5-a840-4bf5-bd2d-5f102fb7b29f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.378269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17d10a5-a840-4bf5-bd2d-5f102fb7b29f admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.379224 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17d10a5-a840-4bf5-bd2d-5f102fb7b29f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.380948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17d10a5-a840-4bf5-bd2d-5f102fb7b29f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.399807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 979/1949] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:58 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:58.401675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75cb8d-19d9-43e3-bc4b-3fdec32b4e2f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.401966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75cb8d-19d9-43e3-bc4b-3fdec32b4e2f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.402395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75cb8d-19d9-43e3-bc4b-3fdec32b4e2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.439673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-0f75cb8d-19d9-43e3-bc4b-3fdec32b4e2f None None] MFA Rules not processed for user `415b09885347444481719e72ecb1475a`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:58.441144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0f75cb8d-19d9-43e3-bc4b-3fdec32b4e2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:58.452732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f81c1-a133-48d2-9cba-349554586202 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.453749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f81c1-a133-48d2-9cba-349554586202 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.454218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f81c1-a133-48d2-9cba-349554586202 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.454681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f81c1-a133-48d2-9cba-349554586202 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.476380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:58.486135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451f81c1-a133-48d2-9cba-349554586202 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.487813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451f81c1-a133-48d2-9cba-349554586202 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.519392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 971/1950] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:41:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:41:58.521258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f81c1-a133-48d2-9cba-349554586202 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.536239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f81c1-a133-48d2-9cba-349554586202 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.542379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d25db1-28d4-486c-907c-f51c3be43667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.544882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d25db1-28d4-486c-907c-f51c3be43667 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.545368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d25db1-28d4-486c-907c-f51c3be43667 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.545849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d25db1-28d4-486c-907c-f51c3be43667 admin admin] PATH_INFO: `/v3/projects/1aa276d907c14a4497a64e154cba47a3/users/53d9f86539ed4d11bd55bf047a073780/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.571505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 980/1951] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:58 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:58.576951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75cb8d-19d9-43e3-bc4b-3fdec32b4e2f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.584336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d25db1-28d4-486c-907c-f51c3be43667 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1aa276d907c14a4497a64e154cba47a3, user_id=53d9f86539ed4d11bd55bf047a073780, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.587372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d25db1-28d4-486c-907c-f51c3be43667 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.603042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75cb8d-19d9-43e3-bc4b-3fdec32b4e2f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.617450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 972/1952] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:58 2022] PUT /identity/v3/projects/1aa276d907c14a4497a64e154cba47a3/users/53d9f86539ed4d11bd55bf047a073780/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:58.617674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9928908-b1d9-4825-b404-5f3f472de5db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.618279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9928908-b1d9-4825-b404-5f3f472de5db admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.618561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9928908-b1d9-4825-b404-5f3f472de5db admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.618864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9928908-b1d9-4825-b404-5f3f472de5db admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.619518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9928908-b1d9-4825-b404-5f3f472de5db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.621984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9928908-b1d9-4825-b404-5f3f472de5db admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.629985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b2639-3f43-4df8-a664-5167224b8b66 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.630312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b2639-3f43-4df8-a664-5167224b8b66 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.630676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b2639-3f43-4df8-a664-5167224b8b66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.674667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3a4b2639-3f43-4df8-a664-5167224b8b66 None None] MFA Rules not processed for user `53d9f86539ed4d11bd55bf047a073780`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:58.676139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3a4b2639-3f43-4df8-a664-5167224b8b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:58.684278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:58.701664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 981/1953] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:58 2022] POST /identity/v3/projects => generated 387 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:58.708161 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9928908-b1d9-4825-b404-5f3f472de5db admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.822596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 973/1954] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:58 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:41:58.823693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b2639-3f43-4df8-a664-5167224b8b66 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:58.899747 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9928908-b1d9-4825-b404-5f3f472de5db admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.917479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eec5a48-4c18-4f50-a78d-82931850de3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:58.923869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eec5a48-4c18-4f50-a78d-82931850de3b admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:58.924966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eec5a48-4c18-4f50-a78d-82931850de3b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:58.928738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eec5a48-4c18-4f50-a78d-82931850de3b admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:58.934124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eec5a48-4c18-4f50-a78d-82931850de3b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:58.940620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eec5a48-4c18-4f50-a78d-82931850de3b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:58.993314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b2639-3f43-4df8-a664-5167224b8b66 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:58.998947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 982/1955] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:58 2022] POST /identity/v3/users => generated 294 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:59.003187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eec5a48-4c18-4f50-a78d-82931850de3b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:59.005682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60551d9e-54dd-4022-8370-a135d3503659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.006408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60551d9e-54dd-4022-8370-a135d3503659 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.006777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60551d9e-54dd-4022-8370-a135d3503659 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.007122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60551d9e-54dd-4022-8370-a135d3503659 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.008358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60551d9e-54dd-4022-8370-a135d3503659 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.013682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60551d9e-54dd-4022-8370-a135d3503659 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.023443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eec5a48-4c18-4f50-a78d-82931850de3b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:59.044473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58118d09-318d-44ea-b387-b068fd1cdcec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.045079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58118d09-318d-44ea-b387-b068fd1cdcec admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.046455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58118d09-318d-44ea-b387-b068fd1cdcec admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.046985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58118d09-318d-44ea-b387-b068fd1cdcec admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.052275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58118d09-318d-44ea-b387-b068fd1cdcec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.054103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58118d09-318d-44ea-b387-b068fd1cdcec admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.072144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 974/1956] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:41:58 2022] POST /identity/v3/projects => generated 381 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:41:59.077645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60551d9e-54dd-4022-8370-a135d3503659 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:59.101187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 983/1957] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:59.102321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58118d09-318d-44ea-b387-b068fd1cdcec admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:59.127575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58118d09-318d-44ea-b387-b068fd1cdcec admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:59.195967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60551d9e-54dd-4022-8370-a135d3503659 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:59.202000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.202656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.203525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.203525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.203883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.206263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.233809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 975/1958] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:41:59 2022] POST /identity/v3/users => generated 297 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:41:59.234690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:59.237342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:59.252898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d49b19-48af-416b-9cad-8f7ed0a56164 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:59.261388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.262766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.263084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.263360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] PATH_INFO: `/v3/projects/1f9434f47212421cbde4ca66664895ca` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.264664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f9434f47212421cbde4ca66664895ca)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.266773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.282787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360194d2-16ad-45cd-af67-94616390743f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.285151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360194d2-16ad-45cd-af67-94616390743f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.285447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360194d2-16ad-45cd-af67-94616390743f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.288136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360194d2-16ad-45cd-af67-94616390743f admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.289063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360194d2-16ad-45cd-af67-94616390743f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.291747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:59.294807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360194d2-16ad-45cd-af67-94616390743f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.331691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f9434f47212421cbde4ca66664895ca', 'request_id': 'req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.333515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 984/1959] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:59.334533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360194d2-16ad-45cd-af67-94616390743f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:59.341064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f9434f47212421cbde4ca66664895ca', 'request_id': 'req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.350432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360194d2-16ad-45cd-af67-94616390743f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:59.351927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f9434f47212421cbde4ca66664895ca', 'request_id': 'req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.370029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:59.371391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] The token cache is being invalidate because project 1f9434f47212421cbde4ca66664895ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:41:59.371883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.372804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.374899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 976/1960] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:41:59 2022] DELETE /identity/v3/projects/1f9434f47212421cbde4ca66664895ca => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:59.376023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:59.377558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:59.479247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.481167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.481633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.482017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a471af-f2be-4dc7-80fa-2a2bf5628fa1 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:59.482847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.483855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.501718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.505749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.507225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.507688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.509020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] PATH_INFO: `/v3/projects/1b1ee70b12a7429ca3c3a8103b79454b/users/48aead6c7470406c983f8a65618d1453/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.527287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 985/1961] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:41:59.528860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:59.531567 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:59.533521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b1ee70b12a7429ca3c3a8103b79454b, user_id=48aead6c7470406c983f8a65618d1453, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.536647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.561263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76569ed0-5208-4aeb-a2bc-0843be6e9feb admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:59.574829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 977/1962] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:59 2022] PUT /identity/v3/projects/1b1ee70b12a7429ca3c3a8103b79454b/users/48aead6c7470406c983f8a65618d1453/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:59.574829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:41:59.574829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:59.705089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a9b83c-c2f3-44c1-bcb8-1b6cdd400a01 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:41:59.711182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc7f682-9e1a-40a0-bedb-6f3efcc40b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.712260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7f682-9e1a-40a0-bedb-6f3efcc40b45 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.712656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7f682-9e1a-40a0-bedb-6f3efcc40b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.713032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7f682-9e1a-40a0-bedb-6f3efcc40b45 admin admin] PATH_INFO: `/v3/projects/e1dfd8d9105f47ee990dfb11a64e5e87/users/369b571282c1403998cae0b758995b53/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.715173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:41:59.715705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.715888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.716173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] PATH_INFO: `/v3/users/98adfeacae964872b4cf423158016da7` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.728373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] RBAC: Authorizing `identity:delete_user(user_id=98adfeacae964872b4cf423158016da7)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.732171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.746027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc7f682-9e1a-40a0-bedb-6f3efcc40b45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1dfd8d9105f47ee990dfb11a64e5e87, user_id=369b571282c1403998cae0b758995b53, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:41:59.750561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc7f682-9e1a-40a0-bedb-6f3efcc40b45 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:41:59.776368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 978/1963] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:41:59 2022] PUT /identity/v3/projects/e1dfd8d9105f47ee990dfb11a64e5e87/users/369b571282c1403998cae0b758995b53/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:41:59.779080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b87ab87-7449-498c-a123-2a9c43907108 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:41:59.779739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b87ab87-7449-498c-a123-2a9c43907108 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:41:59.780791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b87ab87-7449-498c-a123-2a9c43907108 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:41:59.876026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:41:59.909961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98adfeacae964872b4cf423158016da7'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.911788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-0b87ab87-7449-498c-a123-2a9c43907108 None None] MFA Rules not processed for user `48aead6c7470406c983f8a65618d1453`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:41:59.913057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0b87ab87-7449-498c-a123-2a9c43907108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:41:59.928802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98adfeacae964872b4cf423158016da7'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.930861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98adfeacae964872b4cf423158016da7'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.932347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98adfeacae964872b4cf423158016da7'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:41:59.953759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98adfeacae964872b4cf423158016da7'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:00.002272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 986/1964] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:41:59 2022] DELETE /identity/v3/users/98adfeacae964872b4cf423158016da7 => generated 0 bytes in 470 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:00.002272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.015389 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:00.045304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 979/1965] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:41:59 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:00.046949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b87ab87-7449-498c-a123-2a9c43907108 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.048541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0b87ab87-7449-498c-a123-2a9c43907108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:00.104037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:00.174396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96120e8-907b-4d28-9df4-dd1313cb445c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:00.186633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a7e95a-eef8-45b7-97f2-67956230a92b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:00.187789 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7e95a-eef8-45b7-97f2-67956230a92b admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.188646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7e95a-eef8-45b7-97f2-67956230a92b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.190403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7e95a-eef8-45b7-97f2-67956230a92b admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.193308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a7e95a-eef8-45b7-97f2-67956230a92b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:00.197146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a7e95a-eef8-45b7-97f2-67956230a92b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:00.226273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b87ab87-7449-498c-a123-2a9c43907108 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:00.229175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 987/1966] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:00.234359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a7e95a-eef8-45b7-97f2-67956230a92b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.236202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba4f8f-af96-451a-9cfd-4d7f73ca2212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:00.237010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba4f8f-af96-451a-9cfd-4d7f73ca2212 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.237295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba4f8f-af96-451a-9cfd-4d7f73ca2212 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.237576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba4f8f-af96-451a-9cfd-4d7f73ca2212 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.238341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fba4f8f-af96-451a-9cfd-4d7f73ca2212 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:00.240750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fba4f8f-af96-451a-9cfd-4d7f73ca2212 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:00.260250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a7e95a-eef8-45b7-97f2-67956230a92b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:00.280374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee48e1f8-166d-42c5-9880-001c6d672bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:00.281229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee48e1f8-166d-42c5-9880-001c6d672bd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.281613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee48e1f8-166d-42c5-9880-001c6d672bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.281991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee48e1f8-166d-42c5-9880-001c6d672bd1 admin admin] PATH_INFO: `/v3/projects/e1dfd8d9105f47ee990dfb11a64e5e87/users/369b571282c1403998cae0b758995b53/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.284031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 980/1967] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:00 2022] POST /identity/v3/projects => generated 387 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:00.284299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee48e1f8-166d-42c5-9880-001c6d672bd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1dfd8d9105f47ee990dfb11a64e5e87, user_id=369b571282c1403998cae0b758995b53, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:00.287314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee48e1f8-166d-42c5-9880-001c6d672bd1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:00.289033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba4f8f-af96-451a-9cfd-4d7f73ca2212 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.310582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 988/1968] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:00 2022] PUT /identity/v3/projects/e1dfd8d9105f47ee990dfb11a64e5e87/users/369b571282c1403998cae0b758995b53/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:00.324308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cd8c17-a54b-47b7-a0ba-9b4ceffdca10 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.325160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cd8c17-a54b-47b7-a0ba-9b4ceffdca10 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.325412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cd8c17-a54b-47b7-a0ba-9b4ceffdca10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.386151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d8cd8c17-a54b-47b7-a0ba-9b4ceffdca10 None None] MFA Rules not processed for user `369b571282c1403998cae0b758995b53`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:00.390047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d8cd8c17-a54b-47b7-a0ba-9b4ceffdca10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:00.516843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba4f8f-af96-451a-9cfd-4d7f73ca2212 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:00.558649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 989/1969] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:00 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:00.575524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cd8c17-a54b-47b7-a0ba-9b4ceffdca10 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.578037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d8cd8c17-a54b-47b7-a0ba-9b4ceffdca10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:00.599223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:00.647700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:00.648454 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.648817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.649160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.649948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:00.655311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:00.679326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 981/1970] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:00 2022] POST /identity/v3/users => generated 300 bytes in 391 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:00.680411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.683544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:00.684939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cd8c17-a54b-47b7-a0ba-9b4ceffdca10 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:00.689541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f4539-5d9a-4e9d-86eb-79d9c970a1c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:00.690178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f4539-5d9a-4e9d-86eb-79d9c970a1c4 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.690463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f4539-5d9a-4e9d-86eb-79d9c970a1c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.690765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f4539-5d9a-4e9d-86eb-79d9c970a1c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.700773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7967e9-a476-47fa-ab6d-1f267d882904 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:00.701259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5f4539-5d9a-4e9d-86eb-79d9c970a1c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:00.702782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5f4539-5d9a-4e9d-86eb-79d9c970a1c4 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:00.709060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:00.710074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.710359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.710631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] PATH_INFO: `/v3/projects/98aa51eca56642678e3033c2293ac1d1` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.711926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] RBAC: Authorizing `identity:delete_project(project_id=98aa51eca56642678e3033c2293ac1d1)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:00.714244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:00.715605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 990/1971] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:00 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:00.716324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f4539-5d9a-4e9d-86eb-79d9c970a1c4 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.728848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f4539-5d9a-4e9d-86eb-79d9c970a1c4 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:00.751874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf24daa-cb9f-44b3-99be-805030613620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:00.752588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf24daa-cb9f-44b3-99be-805030613620 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.752870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf24daa-cb9f-44b3-99be-805030613620 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.753142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf24daa-cb9f-44b3-99be-805030613620 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.753960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf24daa-cb9f-44b3-99be-805030613620 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:00.759207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf24daa-cb9f-44b3-99be-805030613620 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:00.791414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 991/1972] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:00 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:00.792348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf24daa-cb9f-44b3-99be-805030613620 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.808363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf24daa-cb9f-44b3-99be-805030613620 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:00.839434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98aa51eca56642678e3033c2293ac1d1', 'request_id': 'req-e861c91a-7b69-4d86-bfb0-92d664dfd840'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:00.850300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98aa51eca56642678e3033c2293ac1d1', 'request_id': 'req-e861c91a-7b69-4d86-bfb0-92d664dfd840'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:00.864287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98aa51eca56642678e3033c2293ac1d1', 'request_id': 'req-e861c91a-7b69-4d86-bfb0-92d664dfd840'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:00.873148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:00.889305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] The token cache is being invalidate because project 98aa51eca56642678e3033c2293ac1d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:00.889662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:00.890355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:00.894443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 982/1973] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:00 2022] DELETE /identity/v3/projects/98aa51eca56642678e3033c2293ac1d1 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:00.895449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:00.901428 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:00.935579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44338148-fd01-4777-9e26-04a654dc674a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:00.936608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44338148-fd01-4777-9e26-04a654dc674a admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:00.937071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44338148-fd01-4777-9e26-04a654dc674a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:00.937356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44338148-fd01-4777-9e26-04a654dc674a admin admin] PATH_INFO: `/v3/projects/257061481c1f4bd1a2fc29ab228ebd08/users/b51e1d0ae1fc432ea563c84554fa2efd/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:00.976663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44338148-fd01-4777-9e26-04a654dc674a admin admin] RBAC: Authorizing `identity:create_grant(project_id=257061481c1f4bd1a2fc29ab228ebd08, user_id=b51e1d0ae1fc432ea563c84554fa2efd, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.012577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44338148-fd01-4777-9e26-04a654dc674a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.060191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 992/1974] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:00 2022] PUT /identity/v3/projects/257061481c1f4bd1a2fc29ab228ebd08/users/b51e1d0ae1fc432ea563c84554fa2efd/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:01.067703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d534b9d-4199-4a5a-9f9f-79d965946d9b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.069518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d534b9d-4199-4a5a-9f9f-79d965946d9b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.069814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d534b9d-4199-4a5a-9f9f-79d965946d9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.071094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e861c91a-7b69-4d86-bfb0-92d664dfd840 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.119192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3d534b9d-4199-4a5a-9f9f-79d965946d9b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:01.133396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3d534b9d-4199-4a5a-9f9f-79d965946d9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:01.189415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:01.190217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.191835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.192214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.195334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:01.226174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:01.252641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 993/1975] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:01.253581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d534b9d-4199-4a5a-9f9f-79d965946d9b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:01.255323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3d534b9d-4199-4a5a-9f9f-79d965946d9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:01.269255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d534b9d-4199-4a5a-9f9f-79d965946d9b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.275360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c673cb7-e58e-4b62-8b80-e0fe6bd916c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:01.276165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c673cb7-e58e-4b62-8b80-e0fe6bd916c1 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.276554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c673cb7-e58e-4b62-8b80-e0fe6bd916c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.276935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c673cb7-e58e-4b62-8b80-e0fe6bd916c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.277807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c673cb7-e58e-4b62-8b80-e0fe6bd916c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.279731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c673cb7-e58e-4b62-8b80-e0fe6bd916c1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.301033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 994/1976] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:01 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:01.301033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05036cab-9559-45b9-94d1-84c5c2c9bf71 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.301033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05036cab-9559-45b9-94d1-84c5c2c9bf71 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.303394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05036cab-9559-45b9-94d1-84c5c2c9bf71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.320464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.322251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.336082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-05036cab-9559-45b9-94d1-84c5c2c9bf71 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:01.336986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 983/1977] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:00 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:01.338182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:01.339107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-05036cab-9559-45b9-94d1-84c5c2c9bf71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:01.350162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 995/1978] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:01.352009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c89e03-9584-4a4f-89cd-10093af05ad6 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.356528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05036cab-9559-45b9-94d1-84c5c2c9bf71 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:01.361226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dba8a03-bd70-4d8d-9947-98c091c9b809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:01.362114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba8a03-bd70-4d8d-9947-98c091c9b809 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.362836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba8a03-bd70-4d8d-9947-98c091c9b809 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.363254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba8a03-bd70-4d8d-9947-98c091c9b809 admin admin] PATH_INFO: `/v3/projects/257061481c1f4bd1a2fc29ab228ebd08/users/b51e1d0ae1fc432ea563c84554fa2efd/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.366387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dba8a03-bd70-4d8d-9947-98c091c9b809 admin admin] RBAC: Authorizing `identity:create_grant(project_id=257061481c1f4bd1a2fc29ab228ebd08, user_id=b51e1d0ae1fc432ea563c84554fa2efd, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.369670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05036cab-9559-45b9-94d1-84c5c2c9bf71 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.370722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dba8a03-bd70-4d8d-9947-98c091c9b809 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.375446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:01.376210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.376552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.376833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.377648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:01.377965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.379738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.395028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 984/1979] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:01 2022] PUT /identity/v3/projects/257061481c1f4bd1a2fc29ab228ebd08/users/b51e1d0ae1fc432ea563c84554fa2efd/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:01.407714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404df4b9-b627-4673-ac2c-962d5fcb222c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.408296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404df4b9-b627-4673-ac2c-962d5fcb222c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.408803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404df4b9-b627-4673-ac2c-962d5fcb222c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.412781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 996/1980] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:42:01.413762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:01.425872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a258d-04e7-4065-b10a-7dd39a506d65 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.451049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-404df4b9-b627-4673-ac2c-962d5fcb222c None None] MFA Rules not processed for user `b51e1d0ae1fc432ea563c84554fa2efd`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:01.466188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-404df4b9-b627-4673-ac2c-962d5fcb222c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:01.537532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086a6be8-e658-45d1-a4c2-0dbefb17e357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:01.538207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086a6be8-e658-45d1-a4c2-0dbefb17e357 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.539073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086a6be8-e658-45d1-a4c2-0dbefb17e357 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.539073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086a6be8-e658-45d1-a4c2-0dbefb17e357 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.539536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086a6be8-e658-45d1-a4c2-0dbefb17e357 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.545080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086a6be8-e658-45d1-a4c2-0dbefb17e357 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.591790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 997/1981] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:01 2022] POST /identity/v3/projects => generated 385 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:01.596195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086a6be8-e658-45d1-a4c2-0dbefb17e357 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:01.630559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:01.634331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 985/1982] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:01 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:01.729387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086a6be8-e658-45d1-a4c2-0dbefb17e357 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.735853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf42ac-f77b-446a-b470-8731ee27f853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:01.736980 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf42ac-f77b-446a-b470-8731ee27f853 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.737367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf42ac-f77b-446a-b470-8731ee27f853 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.737923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf42ac-f77b-446a-b470-8731ee27f853 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.738976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf42ac-f77b-446a-b470-8731ee27f853 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.742208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf42ac-f77b-446a-b470-8731ee27f853 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.789438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404df4b9-b627-4673-ac2c-962d5fcb222c None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:01.791490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-404df4b9-b627-4673-ac2c-962d5fcb222c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:01.811082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 998/1983] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:01 2022] POST /identity/v3/users => generated 293 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:01.814787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf42ac-f77b-446a-b470-8731ee27f853 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:01.846260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf42ac-f77b-446a-b470-8731ee27f853 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.860154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7ecccd-737a-4323-bce0-a9ef86e54067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:01.862486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7ecccd-737a-4323-bce0-a9ef86e54067 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.862950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7ecccd-737a-4323-bce0-a9ef86e54067 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.863856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7ecccd-737a-4323-bce0-a9ef86e54067 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.865361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7ecccd-737a-4323-bce0-a9ef86e54067 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.872672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7ecccd-737a-4323-bce0-a9ef86e54067 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.896974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404df4b9-b627-4673-ac2c-962d5fcb222c None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.900738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 999/1984] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:01 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:01.903615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7ecccd-737a-4323-bce0-a9ef86e54067 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:01.925390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7ecccd-737a-4323-bce0-a9ef86e54067 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:01.941885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0633ff23-2170-4bba-a18c-862d87761f7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:01.942523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0633ff23-2170-4bba-a18c-862d87761f7c admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:01.946960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0633ff23-2170-4bba-a18c-862d87761f7c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:01.947304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0633ff23-2170-4bba-a18c-862d87761f7c admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:01.952590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0633ff23-2170-4bba-a18c-862d87761f7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:01.957373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0633ff23-2170-4bba-a18c-862d87761f7c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:01.987147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1000/1985] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:01 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:01.989135 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0633ff23-2170-4bba-a18c-862d87761f7c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:02.009236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0633ff23-2170-4bba-a18c-862d87761f7c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:02.012421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:02.022702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d073d32-207a-4cae-aade-a907ef537c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:02.023524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d073d32-207a-4cae-aade-a907ef537c87 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.023838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d073d32-207a-4cae-aade-a907ef537c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.024087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d073d32-207a-4cae-aade-a907ef537c87 admin admin] PATH_INFO: `/v3/projects/359d9786f78e4f75aa482824de41d3ce/users/052f39b83f3e49dcac6267ff61f30060/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:02.037215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:02.038194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.038675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.039201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:02.052832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d073d32-207a-4cae-aade-a907ef537c87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=359d9786f78e4f75aa482824de41d3ce, user_id=052f39b83f3e49dcac6267ff61f30060, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:02.056527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d073d32-207a-4cae-aade-a907ef537c87 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:02.088046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1001/1986] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:01 2022] PUT /identity/v3/projects/359d9786f78e4f75aa482824de41d3ce/users/052f39b83f3e49dcac6267ff61f30060/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:02.099648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d6c55-8431-4c67-876c-8e0be8832d86 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.100038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d6c55-8431-4c67-876c-8e0be8832d86 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.100620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d6c55-8431-4c67-876c-8e0be8832d86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:02.140649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:02.142187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:02.151872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d64d6c55-8431-4c67-876c-8e0be8832d86 None None] MFA Rules not processed for user `052f39b83f3e49dcac6267ff61f30060`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:02.153451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d64d6c55-8431-4c67-876c-8e0be8832d86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:02.155236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 986/1987] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:01 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:02.156247 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:02.158302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:02.187566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b845ff57-bd36-40f8-9dbb-694981d955c2 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:02.265855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1002/1988] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:02.288117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:02.289970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.292505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64d6c55-8431-4c67-876c-8e0be8832d86 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:02.298975 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.299576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] PATH_INFO: `/v3/users/53d9f86539ed4d11bd55bf047a073780` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:02.304256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] RBAC: Authorizing `identity:delete_user(user_id=53d9f86539ed4d11bd55bf047a073780)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:02.317209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:02.411468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:02.422241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53d9f86539ed4d11bd55bf047a073780'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.423716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53d9f86539ed4d11bd55bf047a073780'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.432536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53d9f86539ed4d11bd55bf047a073780'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.432746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64d6c55-8431-4c67-876c-8e0be8832d86 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:02.444420 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53d9f86539ed4d11bd55bf047a073780'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.447111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4371a54-a784-4f0f-ad13-c67a823c6a1c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:02.447969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4371a54-a784-4f0f-ad13-c67a823c6a1c service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.448709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4371a54-a784-4f0f-ad13-c67a823c6a1c service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.449072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4371a54-a784-4f0f-ad13-c67a823c6a1c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:02.461482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4371a54-a784-4f0f-ad13-c67a823c6a1c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:02.462446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-06367dac-ec5a-4b91-963f-35eebd24e32e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53d9f86539ed4d11bd55bf047a073780'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.463909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4371a54-a784-4f0f-ad13-c67a823c6a1c service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:02.476507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 987/1989] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:02 2022] DELETE /identity/v3/users/53d9f86539ed4d11bd55bf047a073780 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:02.482031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd90c8d8-05d6-4bcb-b226-8a7da1ef9f48 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.482408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd90c8d8-05d6-4bcb-b226-8a7da1ef9f48 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.483282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd90c8d8-05d6-4bcb-b226-8a7da1ef9f48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:02.486449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1003/1990] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:02 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:02.520702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4371a54-a784-4f0f-ad13-c67a823c6a1c service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:02.531322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-fd90c8d8-05d6-4bcb-b226-8a7da1ef9f48 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:02.532896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fd90c8d8-05d6-4bcb-b226-8a7da1ef9f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:02.542932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:02.553444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4371a54-a784-4f0f-ad13-c67a823c6a1c service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:02.678183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 988/1991] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:02.689617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd90c8d8-05d6-4bcb-b226-8a7da1ef9f48 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:02.691215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fd90c8d8-05d6-4bcb-b226-8a7da1ef9f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:02.770156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:02.772884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.773659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.775738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] PATH_INFO: `/v3/users/052f39b83f3e49dcac6267ff61f30060` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:02.777300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] RBAC: Authorizing `identity:delete_user(user_id=052f39b83f3e49dcac6267ff61f30060)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:02.788221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:02.810115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd90c8d8-05d6-4bcb-b226-8a7da1ef9f48 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:02.822648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bb25ad-048d-4a34-a6a6-280c71ae06e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:02.823816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb25ad-048d-4a34-a6a6-280c71ae06e3 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.824101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb25ad-048d-4a34-a6a6-280c71ae06e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.827521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb25ad-048d-4a34-a6a6-280c71ae06e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:02.852501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bb25ad-048d-4a34-a6a6-280c71ae06e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:02.854780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bb25ad-048d-4a34-a6a6-280c71ae06e3 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:02.891682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '052f39b83f3e49dcac6267ff61f30060'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.900592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '052f39b83f3e49dcac6267ff61f30060'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.901118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 989/1992] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:02.901321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '052f39b83f3e49dcac6267ff61f30060'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.902130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '052f39b83f3e49dcac6267ff61f30060'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.915911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '052f39b83f3e49dcac6267ff61f30060'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:02.969869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1004/1993] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:02 2022] DELETE /identity/v3/users/052f39b83f3e49dcac6267ff61f30060 => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:02.973883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabaf6bb-caef-4aba-a182-eb07868736c0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:02.974929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabaf6bb-caef-4aba-a182-eb07868736c0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:02.975715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabaf6bb-caef-4aba-a182-eb07868736c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:03.033171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-dabaf6bb-caef-4aba-a182-eb07868736c0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:03.035737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-dabaf6bb-caef-4aba-a182-eb07868736c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:03.170225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 990/1994] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:03.192885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:03.197363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:03.329705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabaf6bb-caef-4aba-a182-eb07868736c0 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:03.334495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-dabaf6bb-caef-4aba-a182-eb07868736c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:03.354062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f667e9b-6e7c-4091-8fa6-c6c55c20c588 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:03.369833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:03.372340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:03.374173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:03.374980 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:03.398453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:03.405653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:03.447848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1005/1995] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:03.503665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabaf6bb-caef-4aba-a182-eb07868736c0 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:03.508173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:03.508801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:03.509095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:03.509395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:03.511451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:03.526532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:03.622108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:03.623703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:03.652698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 991/1996] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:03 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:03.666378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:03.667865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:03.704148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac80b5a0-7709-4f91-94d7-7e69a7337491 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:03.711060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:03.711720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:03.712004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:03.712278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] PATH_INFO: `/v3/projects/1aa276d907c14a4497a64e154cba47a3` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:03.714185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1aa276d907c14a4497a64e154cba47a3)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:03.717169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:03.736969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:03.843425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:03.846584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:03.859827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1aa276d907c14a4497a64e154cba47a3', 'request_id': 'req-236c5f58-e450-4409-bf7d-db80a100db53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:03.873237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1aa276d907c14a4497a64e154cba47a3', 'request_id': 'req-236c5f58-e450-4409-bf7d-db80a100db53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:03.874259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3884fc-4a46-4653-8e82-6c65637e8c4d service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:03.901744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1aa276d907c14a4497a64e154cba47a3', 'request_id': 'req-236c5f58-e450-4409-bf7d-db80a100db53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:03.921217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] The token cache is being invalidate because project 1aa276d907c14a4497a64e154cba47a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:03.921521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:03.922346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:03.924252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1006/1997] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:03 2022] DELETE /identity/v3/projects/1aa276d907c14a4497a64e154cba47a3 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:03.932647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:03.937496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:04.031386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.032403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.035466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.035841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] PATH_INFO: `/v3/projects/359d9786f78e4f75aa482824de41d3ce` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.041061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] RBAC: Authorizing `identity:delete_project(project_id=359d9786f78e4f75aa482824de41d3ce)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.056939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.059870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236c5f58-e450-4409-bf7d-db80a100db53 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.065620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.070155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.070471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.070799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] PATH_INFO: `/v3/users/369b571282c1403998cae0b758995b53` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.072409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=369b571282c1403998cae0b758995b53)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.076989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.156008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '369b571282c1403998cae0b758995b53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.169560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '369b571282c1403998cae0b758995b53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.170974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '369b571282c1403998cae0b758995b53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.171920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '369b571282c1403998cae0b758995b53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.186202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-58fa05c8-2965-4067-81fe-dcad2263a9d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '369b571282c1403998cae0b758995b53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.187479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '359d9786f78e4f75aa482824de41d3ce', 'request_id': 'req-8d19b58e-4f14-4932-8a5f-15d34840b468'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.196464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '359d9786f78e4f75aa482824de41d3ce', 'request_id': 'req-8d19b58e-4f14-4932-8a5f-15d34840b468'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.206232 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '359d9786f78e4f75aa482824de41d3ce', 'request_id': 'req-8d19b58e-4f14-4932-8a5f-15d34840b468'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.217979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] The token cache is being invalidate because project 359d9786f78e4f75aa482824de41d3ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:04.218325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.219018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d19b58e-4f14-4932-8a5f-15d34840b468 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.221058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 992/1998] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:03 2022] DELETE /identity/v3/projects/359d9786f78e4f75aa482824de41d3ce => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:04.227453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e8bbd6-b7f2-4d9f-b49e-d9673056405c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.227828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e8bbd6-b7f2-4d9f-b49e-d9673056405c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.228166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e8bbd6-b7f2-4d9f-b49e-d9673056405c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.228335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1007/1999] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:03 2022] DELETE /identity/v3/users/369b571282c1403998cae0b758995b53 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:04.250013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-02e8bbd6-b7f2-4d9f-b49e-d9673056405c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:04.251340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-02e8bbd6-b7f2-4d9f-b49e-d9673056405c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:04.304794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:04.333497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 993/2000] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:04.344006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359736fa-d057-457e-8e12-1a3d0851c0fe None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.344722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359736fa-d057-457e-8e12-1a3d0851c0fe None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.345114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359736fa-d057-457e-8e12-1a3d0851c0fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.365249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-359736fa-d057-457e-8e12-1a3d0851c0fe None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:04.367008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-359736fa-d057-457e-8e12-1a3d0851c0fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:04.374673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1008/2001] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:04.378957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e8bbd6-b7f2-4d9f-b49e-d9673056405c None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:04.390295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e8bbd6-b7f2-4d9f-b49e-d9673056405c None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.411361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.411918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.412195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.412466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.413162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:04.413473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.415167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.429379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 994/2002] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:42:04.433569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359736fa-d057-457e-8e12-1a3d0851c0fe None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:04.441037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:04.445711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359736fa-d057-457e-8e12-1a3d0851c0fe None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.453017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.453823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.454182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.454536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.455319 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.457540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.491870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1009/2003] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:04 2022] POST /identity/v3/projects => generated 375 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:04.493159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:04.505568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7dbc78-5684-494c-9037-c0300fc224f9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.529007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:04.530792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:04.575250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edab6ef7-cd24-438a-b7b3-612d2e8c4fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.575873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab6ef7-cd24-438a-b7b3-612d2e8c4fd2 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.576155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab6ef7-cd24-438a-b7b3-612d2e8c4fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.576430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab6ef7-cd24-438a-b7b3-612d2e8c4fd2 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.577064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edab6ef7-cd24-438a-b7b3-612d2e8c4fd2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.580137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edab6ef7-cd24-438a-b7b3-612d2e8c4fd2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.603473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 995/2004] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:04 2022] POST /identity/v3/users => generated 288 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:04.607312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edab6ef7-cd24-438a-b7b3-612d2e8c4fd2 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:04.617278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edab6ef7-cd24-438a-b7b3-612d2e8c4fd2 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.622238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cb171c-3803-4f7c-9fa0-f9f5c16165fe admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.622468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747ee316-2267-4867-813b-8811c5e4abe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.623110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747ee316-2267-4867-813b-8811c5e4abe4 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.623389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747ee316-2267-4867-813b-8811c5e4abe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.623661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747ee316-2267-4867-813b-8811c5e4abe4 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.624309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747ee316-2267-4867-813b-8811c5e4abe4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.625971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747ee316-2267-4867-813b-8811c5e4abe4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.627818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.628886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.629175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.629454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] PATH_INFO: `/v3/projects/e1dfd8d9105f47ee990dfb11a64e5e87` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.630621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1dfd8d9105f47ee990dfb11a64e5e87)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.633114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.637111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 996/2005] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:04.641154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747ee316-2267-4867-813b-8811c5e4abe4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:04.651897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747ee316-2267-4867-813b-8811c5e4abe4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.657476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dc8cdf-9c85-4980-ad2d-2237354a8b16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.658102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dc8cdf-9c85-4980-ad2d-2237354a8b16 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.658371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dc8cdf-9c85-4980-ad2d-2237354a8b16 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.658646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dc8cdf-9c85-4980-ad2d-2237354a8b16 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.659333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dc8cdf-9c85-4980-ad2d-2237354a8b16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.660927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dc8cdf-9c85-4980-ad2d-2237354a8b16 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.675972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 997/2006] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:04.677066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dc8cdf-9c85-4980-ad2d-2237354a8b16 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:04.684892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1dfd8d9105f47ee990dfb11a64e5e87', 'request_id': 'req-66e35a54-f66a-4aaa-b354-4514b318d443'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.688243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dc8cdf-9c85-4980-ad2d-2237354a8b16 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.691656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1dfd8d9105f47ee990dfb11a64e5e87', 'request_id': 'req-66e35a54-f66a-4aaa-b354-4514b318d443'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.697355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1dfd8d9105f47ee990dfb11a64e5e87', 'request_id': 'req-66e35a54-f66a-4aaa-b354-4514b318d443'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.711125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] The token cache is being invalidate because project e1dfd8d9105f47ee990dfb11a64e5e87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:04.711422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.712207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-66e35a54-f66a-4aaa-b354-4514b318d443 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:04.713817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1010/2007] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:04 2022] DELETE /identity/v3/projects/e1dfd8d9105f47ee990dfb11a64e5e87 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:04.721273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec1b57-8daf-45c6-b650-607e2e0b0a3e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.721694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec1b57-8daf-45c6-b650-607e2e0b0a3e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.722084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec1b57-8daf-45c6-b650-607e2e0b0a3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.749726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-10ec1b57-8daf-45c6-b650-607e2e0b0a3e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:04.751183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-10ec1b57-8daf-45c6-b650-607e2e0b0a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:04.764095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f4848-eadb-49a8-98ac-80e22a39c23f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.764960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f4848-eadb-49a8-98ac-80e22a39c23f admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.765231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f4848-eadb-49a8-98ac-80e22a39c23f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.765857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f4848-eadb-49a8-98ac-80e22a39c23f admin admin] PATH_INFO: `/v3/projects/45ed2afc7ed94581afedc9ac52eb9599/users/37285b86b6664472925ac1c12fba1301/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.791363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33f4848-eadb-49a8-98ac-80e22a39c23f admin admin] RBAC: Authorizing `identity:create_grant(project_id=45ed2afc7ed94581afedc9ac52eb9599, user_id=37285b86b6664472925ac1c12fba1301, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.794774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33f4848-eadb-49a8-98ac-80e22a39c23f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:04.826655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 998/2008] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:04 2022] PUT /identity/v3/projects/45ed2afc7ed94581afedc9ac52eb9599/users/37285b86b6664472925ac1c12fba1301/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:04.834687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1011/2009] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:04.841494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa1e6f1-46eb-408b-8b67-35ccfc6ee650 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.842591 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa1e6f1-46eb-408b-8b67-35ccfc6ee650 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.844863 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9b9700-058a-4192-b67c-2af3c0010f35 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.845198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9b9700-058a-4192-b67c-2af3c0010f35 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.845477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9b9700-058a-4192-b67c-2af3c0010f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.846833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa1e6f1-46eb-408b-8b67-35ccfc6ee650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.869568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-cb9b9700-058a-4192-b67c-2af3c0010f35 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:04.870992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cb9b9700-058a-4192-b67c-2af3c0010f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:04.882211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:04.889343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-caa1e6f1-46eb-408b-8b67-35ccfc6ee650 None None] MFA Rules not processed for user `37285b86b6664472925ac1c12fba1301`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:04.890636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-caa1e6f1-46eb-408b-8b67-35ccfc6ee650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:04.903715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:04.951878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1012/2010] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:04.955746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9b9700-058a-4192-b67c-2af3c0010f35 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:04.970141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9b9700-058a-4192-b67c-2af3c0010f35 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:04.976084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:04.976899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:04.977292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:04.978275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:04.979112 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:04.980088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:04.982653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.000668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1013/2011] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:42:05.004706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.009943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 999/2012] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:04 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:05.014965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa1e6f1-46eb-408b-8b67-35ccfc6ee650 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.016334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-caa1e6f1-46eb-408b-8b67-35ccfc6ee650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:05.019886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b9abaf-5a59-4bf5-8c31-66475d228559 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.025420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eff92d-aa01-4787-af80-07cf3b80d7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.026162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eff92d-aa01-4787-af80-07cf3b80d7df admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.026449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eff92d-aa01-4787-af80-07cf3b80d7df admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.026769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eff92d-aa01-4787-af80-07cf3b80d7df admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.027475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6eff92d-aa01-4787-af80-07cf3b80d7df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.029945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6eff92d-aa01-4787-af80-07cf3b80d7df admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.031850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa1e6f1-46eb-408b-8b67-35ccfc6ee650 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.036462 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af772fc1-ca3e-4b04-bac2-3e68b4be8de3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.037186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af772fc1-ca3e-4b04-bac2-3e68b4be8de3 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.037530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af772fc1-ca3e-4b04-bac2-3e68b4be8de3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.037863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af772fc1-ca3e-4b04-bac2-3e68b4be8de3 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.038569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af772fc1-ca3e-4b04-bac2-3e68b4be8de3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.040582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af772fc1-ca3e-4b04-bac2-3e68b4be8de3 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.060888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1014/2013] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:05 2022] POST /identity/v3/projects => generated 375 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:05.065089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eff92d-aa01-4787-af80-07cf3b80d7df admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.072345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1000/2014] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:05 2022] POST /identity/v3/projects => generated 375 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:05.077217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af772fc1-ca3e-4b04-bac2-3e68b4be8de3 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.139414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eff92d-aa01-4787-af80-07cf3b80d7df admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.160421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af772fc1-ca3e-4b04-bac2-3e68b4be8de3 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.165578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0321f5bc-acc3-48be-8d52-d2298a4c3ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.166345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0321f5bc-acc3-48be-8d52-d2298a4c3ce9 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.166624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0321f5bc-acc3-48be-8d52-d2298a4c3ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.167019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0321f5bc-acc3-48be-8d52-d2298a4c3ce9 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.167772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0321f5bc-acc3-48be-8d52-d2298a4c3ce9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.170376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0321f5bc-acc3-48be-8d52-d2298a4c3ce9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.202108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1001/2015] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:05 2022] POST /identity/v3/users => generated 294 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:05.207628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0321f5bc-acc3-48be-8d52-d2298a4c3ce9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.228206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0321f5bc-acc3-48be-8d52-d2298a4c3ce9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.233773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965f498-96d7-4d7a-bac0-ca68e7d13917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.234948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965f498-96d7-4d7a-bac0-ca68e7d13917 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.234948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965f498-96d7-4d7a-bac0-ca68e7d13917 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.235662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2965f498-96d7-4d7a-bac0-ca68e7d13917 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.236225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f4e8eb-9290-4d8f-8974-bbeb838ec5f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.236895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f4e8eb-9290-4d8f-8974-bbeb838ec5f6 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.237178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f4e8eb-9290-4d8f-8974-bbeb838ec5f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.237515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f4e8eb-9290-4d8f-8974-bbeb838ec5f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.237628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2965f498-96d7-4d7a-bac0-ca68e7d13917 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.238371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f4e8eb-9290-4d8f-8974-bbeb838ec5f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.239919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f4e8eb-9290-4d8f-8974-bbeb838ec5f6 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.242119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2965f498-96d7-4d7a-bac0-ca68e7d13917 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.255557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1002/2016] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:05 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:05.261205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f4e8eb-9290-4d8f-8974-bbeb838ec5f6 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.281082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1015/2017] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:05 2022] POST /identity/v3/users => generated 288 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:05.287658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f4e8eb-9290-4d8f-8974-bbeb838ec5f6 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.291570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965f498-96d7-4d7a-bac0-ca68e7d13917 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.297592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a58c4c-b1df-4020-aa1d-84a30526bfe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.298782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a58c4c-b1df-4020-aa1d-84a30526bfe4 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.299565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a58c4c-b1df-4020-aa1d-84a30526bfe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.300027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a58c4c-b1df-4020-aa1d-84a30526bfe4 admin admin] PATH_INFO: `/v3/projects/27369164e35c451ea75e9fff08bf4d43/users/95035303830c4e6aaf5b58895e581a47/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.311729 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2965f498-96d7-4d7a-bac0-ca68e7d13917 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.317917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2033ffb-6813-45ba-b4e5-ca06bac4653d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.318564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2033ffb-6813-45ba-b4e5-ca06bac4653d admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.318873 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2033ffb-6813-45ba-b4e5-ca06bac4653d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.319157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2033ffb-6813-45ba-b4e5-ca06bac4653d admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.319794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2033ffb-6813-45ba-b4e5-ca06bac4653d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.321963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2033ffb-6813-45ba-b4e5-ca06bac4653d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.332662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a58c4c-b1df-4020-aa1d-84a30526bfe4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27369164e35c451ea75e9fff08bf4d43, user_id=95035303830c4e6aaf5b58895e581a47, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.335494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1016/2018] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:05 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:05.341463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2033ffb-6813-45ba-b4e5-ca06bac4653d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.342840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a58c4c-b1df-4020-aa1d-84a30526bfe4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.362486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:05.373371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2033ffb-6813-45ba-b4e5-ca06bac4653d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.379955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1003/2019] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:05 2022] PUT /identity/v3/projects/27369164e35c451ea75e9fff08bf4d43/users/95035303830c4e6aaf5b58895e581a47/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:05.384192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a58c4c-b1df-4020-aa1d-84a30526bfe4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.453812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:05.465760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437b8c1-314d-465e-8614-73fd43c258fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.466488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437b8c1-314d-465e-8614-73fd43c258fc admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.466897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437b8c1-314d-465e-8614-73fd43c258fc admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.467239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437b8c1-314d-465e-8614-73fd43c258fc admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.467964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437b8c1-314d-465e-8614-73fd43c258fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.472272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437b8c1-314d-465e-8614-73fd43c258fc admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.487273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1017/2020] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:05 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:05.492968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437b8c1-314d-465e-8614-73fd43c258fc admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.509111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437b8c1-314d-465e-8614-73fd43c258fc admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.509373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a58c4c-b1df-4020-aa1d-84a30526bfe4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.514835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd4f5fc-98e3-4105-ae20-a2196e438cce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.515576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4f5fc-98e3-4105-ae20-a2196e438cce admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.515913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4f5fc-98e3-4105-ae20-a2196e438cce admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.516408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4f5fc-98e3-4105-ae20-a2196e438cce admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.517168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd4f5fc-98e3-4105-ae20-a2196e438cce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.518892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd4f5fc-98e3-4105-ae20-a2196e438cce admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.529054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca15e5c0-4523-4286-aa7f-1d34f2e6ff28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.529874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca15e5c0-4523-4286-aa7f-1d34f2e6ff28 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.530150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca15e5c0-4523-4286-aa7f-1d34f2e6ff28 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.530419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca15e5c0-4523-4286-aa7f-1d34f2e6ff28 admin admin] PATH_INFO: `/v3/projects/f3b80a1e2fcc414bb50798dfd6571d99/users/2aa0cdf3ca35427ab0cf4401afc6e65e/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.534149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1004/2021] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:05 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:05.538858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd4f5fc-98e3-4105-ae20-a2196e438cce admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.549566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca15e5c0-4523-4286-aa7f-1d34f2e6ff28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3b80a1e2fcc414bb50798dfd6571d99, user_id=2aa0cdf3ca35427ab0cf4401afc6e65e, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.552294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca15e5c0-4523-4286-aa7f-1d34f2e6ff28 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.557752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd4f5fc-98e3-4105-ae20-a2196e438cce admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.563531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61205f9b-2a69-47a2-b73f-55b24576b0a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.564527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61205f9b-2a69-47a2-b73f-55b24576b0a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.565115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61205f9b-2a69-47a2-b73f-55b24576b0a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.565609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61205f9b-2a69-47a2-b73f-55b24576b0a5 admin admin] PATH_INFO: `/v3/projects/27369164e35c451ea75e9fff08bf4d43/users/95035303830c4e6aaf5b58895e581a47/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.568450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61205f9b-2a69-47a2-b73f-55b24576b0a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27369164e35c451ea75e9fff08bf4d43, user_id=95035303830c4e6aaf5b58895e581a47, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.570839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61205f9b-2a69-47a2-b73f-55b24576b0a5 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.574757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1018/2022] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:05 2022] PUT /identity/v3/projects/f3b80a1e2fcc414bb50798dfd6571d99/users/2aa0cdf3ca35427ab0cf4401afc6e65e/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:05.585292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1005/2023] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:05 2022] PUT /identity/v3/projects/27369164e35c451ea75e9fff08bf4d43/users/95035303830c4e6aaf5b58895e581a47/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:05.587295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb396f-2ff3-4c6f-825e-d628a4deb275 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.587717 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb396f-2ff3-4c6f-825e-d628a4deb275 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.588057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb396f-2ff3-4c6f-825e-d628a4deb275 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.597044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f848-13b7-4070-a15e-f288def78ab7 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.597409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f848-13b7-4070-a15e-f288def78ab7 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.597694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f848-13b7-4070-a15e-f288def78ab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.626493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e2bb396f-2ff3-4c6f-825e-d628a4deb275 None None] MFA Rules not processed for user `2aa0cdf3ca35427ab0cf4401afc6e65e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:05.628415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e2bb396f-2ff3-4c6f-825e-d628a4deb275 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:05.631947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-3d13f848-13b7-4070-a15e-f288def78ab7 None None] MFA Rules not processed for user `95035303830c4e6aaf5b58895e581a47`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:05.633154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3d13f848-13b7-4070-a15e-f288def78ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:05.726398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1019/2024] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:05 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:05.730864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1006/2025] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:05 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:05.738793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb396f-2ff3-4c6f-825e-d628a4deb275 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.739267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d13f848-13b7-4070-a15e-f288def78ab7 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:05.740437 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e2bb396f-2ff3-4c6f-825e-d628a4deb275 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:05.741241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3d13f848-13b7-4070-a15e-f288def78ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:05.808351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d13f848-13b7-4070-a15e-f288def78ab7 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.813203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6d46d-34bf-4a07-acb7-449ddccd1ff5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.813366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb396f-2ff3-4c6f-825e-d628a4deb275 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:05.813827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6d46d-34bf-4a07-acb7-449ddccd1ff5 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.814104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6d46d-34bf-4a07-acb7-449ddccd1ff5 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.814377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6d46d-34bf-4a07-acb7-449ddccd1ff5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.818905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a08d8d5-1e0a-4517-8eb3-7afb0af01d71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:05.819835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a08d8d5-1e0a-4517-8eb3-7afb0af01d71 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:05.820340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a08d8d5-1e0a-4517-8eb3-7afb0af01d71 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:05.820822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a08d8d5-1e0a-4517-8eb3-7afb0af01d71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:05.833511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a08d8d5-1e0a-4517-8eb3-7afb0af01d71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.835012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a08d8d5-1e0a-4517-8eb3-7afb0af01d71 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.851609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1020/2026] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:05 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:05.909400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d6d46d-34bf-4a07-acb7-449ddccd1ff5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:05.913286 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d6d46d-34bf-4a07-acb7-449ddccd1ff5 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:05.942181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1007/2027] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:05 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:08.141841 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a08d8d5-1e0a-4517-8eb3-7afb0af01d71 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:08.151467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a08d8d5-1e0a-4517-8eb3-7afb0af01d71 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:08.157208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9fe8a6-26c5-422d-9fa5-5ce2a74e2192 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:08.158235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9fe8a6-26c5-422d-9fa5-5ce2a74e2192 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:08.159205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9fe8a6-26c5-422d-9fa5-5ce2a74e2192 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:08.159586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9fe8a6-26c5-422d-9fa5-5ce2a74e2192 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:08.162243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-be9fe8a6-26c5-422d-9fa5-5ce2a74e2192 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:08.177251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:08.264680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9fe8a6-26c5-422d-9fa5-5ce2a74e2192 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:08.266055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9fe8a6-26c5-422d-9fa5-5ce2a74e2192 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:08.276455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1021/2028] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:08 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:08.853023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6d46d-34bf-4a07-acb7-449ddccd1ff5 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:08.860999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:08.929332 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6d46d-34bf-4a07-acb7-449ddccd1ff5 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:08.936399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:08.938200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:08.938200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:08.938200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] PATH_INFO: `/v3/users/37285b86b6664472925ac1c12fba1301` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:08.939277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] RBAC: Authorizing `identity:delete_user(user_id=37285b86b6664472925ac1c12fba1301)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:08.941790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:09.016446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37285b86b6664472925ac1c12fba1301'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.017895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37285b86b6664472925ac1c12fba1301'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.030543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37285b86b6664472925ac1c12fba1301'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.040624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37285b86b6664472925ac1c12fba1301'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.056516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37285b86b6664472925ac1c12fba1301'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.070303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1008/2029] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:08 2022] DELETE /identity/v3/users/37285b86b6664472925ac1c12fba1301 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:09.072013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c4451-2d2e-43f3-9788-038e8b571ca6 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:09.072555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c4451-2d2e-43f3-9788-038e8b571ca6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:09.072831 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c4451-2d2e-43f3-9788-038e8b571ca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:09.095719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-ec6c4451-2d2e-43f3-9788-038e8b571ca6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:09.096982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ec6c4451-2d2e-43f3-9788-038e8b571ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:09.182329 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1022/2030] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:09.193409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:09.195562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:09.276285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da251a0-d5d4-45b8-b372-11069d60207d admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:09.281045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77702dac-95b0-4a2c-a2fc-9ae8e2c46ea0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:09.281764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77702dac-95b0-4a2c-a2fc-9ae8e2c46ea0 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:09.282040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77702dac-95b0-4a2c-a2fc-9ae8e2c46ea0 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:09.282600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77702dac-95b0-4a2c-a2fc-9ae8e2c46ea0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:09.303114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77702dac-95b0-4a2c-a2fc-9ae8e2c46ea0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:09.304997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77702dac-95b0-4a2c-a2fc-9ae8e2c46ea0 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:09.318040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1009/2031] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:09.611653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6c4451-2d2e-43f3-9788-038e8b571ca6 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:09.613300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ec6c4451-2d2e-43f3-9788-038e8b571ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:09.631369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6c4451-2d2e-43f3-9788-038e8b571ca6 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:09.636044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:09.637199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:09.637508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:09.637782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] PATH_INFO: `/v3/projects/45ed2afc7ed94581afedc9ac52eb9599` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:09.638943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=45ed2afc7ed94581afedc9ac52eb9599)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:09.640846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:09.701987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45ed2afc7ed94581afedc9ac52eb9599', 'request_id': 'req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.708633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45ed2afc7ed94581afedc9ac52eb9599', 'request_id': 'req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.714424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45ed2afc7ed94581afedc9ac52eb9599', 'request_id': 'req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.729327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] The token cache is being invalidate because project 45ed2afc7ed94581afedc9ac52eb9599 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:09.729618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.730264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.732032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1023/2032] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:09 2022] DELETE /identity/v3/projects/45ed2afc7ed94581afedc9ac52eb9599 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:09.737441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77702dac-95b0-4a2c-a2fc-9ae8e2c46ea0 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:09.751578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77702dac-95b0-4a2c-a2fc-9ae8e2c46ea0 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:09.778277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:09.831504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:09.833704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:09.834011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:09.834294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] PATH_INFO: `/v3/users/95035303830c4e6aaf5b58895e581a47` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:09.835563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=95035303830c4e6aaf5b58895e581a47)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:09.837553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:09.903594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95035303830c4e6aaf5b58895e581a47'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.904385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95035303830c4e6aaf5b58895e581a47'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.912350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95035303830c4e6aaf5b58895e581a47'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.928060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95035303830c4e6aaf5b58895e581a47'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.947769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b2891263-b10c-4421-a698-59354c46eb2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95035303830c4e6aaf5b58895e581a47'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:09.956134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1010/2033] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:09 2022] DELETE /identity/v3/users/95035303830c4e6aaf5b58895e581a47 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:10.378370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:10.379847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:10.401050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d559c8a-3dc0-48cb-bfa4-169d98b16b11 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:10.485669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:10.546703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:10.547606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:10.547888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:10.551411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] PATH_INFO: `/v3/projects/27369164e35c451ea75e9fff08bf4d43` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:10.553113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27369164e35c451ea75e9fff08bf4d43)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:10.556555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:10.641109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27369164e35c451ea75e9fff08bf4d43', 'request_id': 'req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:10.653555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27369164e35c451ea75e9fff08bf4d43', 'request_id': 'req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:10.662135 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27369164e35c451ea75e9fff08bf4d43', 'request_id': 'req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:10.685346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] The token cache is being invalidate because project 27369164e35c451ea75e9fff08bf4d43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:10.685667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:10.686340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-07b45461-4705-4a09-a9f6-1cf3a54a0cc3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:10.688713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1024/2034] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:10 2022] DELETE /identity/v3/projects/27369164e35c451ea75e9fff08bf4d43 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:10.694482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d021d-41ff-4ebe-925d-e1563753923f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:10.694482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d021d-41ff-4ebe-925d-e1563753923f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:10.694482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d021d-41ff-4ebe-925d-e1563753923f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:10.720521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-857d021d-41ff-4ebe-925d-e1563753923f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:10.722241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-857d021d-41ff-4ebe-925d-e1563753923f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:10.837386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1011/2035] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:10.844860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a299f944-3017-4b45-ba63-d940420e53cf None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:10.845154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a299f944-3017-4b45-ba63-d940420e53cf None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:10.845557 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a299f944-3017-4b45-ba63-d940420e53cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:10.874226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a299f944-3017-4b45-ba63-d940420e53cf None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:10.876625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a299f944-3017-4b45-ba63-d940420e53cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:10.892776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1025/2036] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:10.896413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d021d-41ff-4ebe-925d-e1563753923f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:10.926167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d021d-41ff-4ebe-925d-e1563753923f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:10.939411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a299f944-3017-4b45-ba63-d940420e53cf None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:10.941479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a299f944-3017-4b45-ba63-d940420e53cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:10.945155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:10.946555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:10.947571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:10.948345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:10.950391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:10.951306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:10.954880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:10.960019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a299f944-3017-4b45-ba63-d940420e53cf None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:10.968441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:10.970382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:10.970883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:10.972115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] PATH_INFO: `/v3/users/415b09885347444481719e72ecb1475a` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:10.975203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=415b09885347444481719e72ecb1475a)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:10.977864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:10.978528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1012/2037] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:42:10.982665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:10.994682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e025f8a-d309-4f21-82ae-b776d3a94e3f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:11.000578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:11.001453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.001894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.002322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.003184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:11.017641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:11.064323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '415b09885347444481719e72ecb1475a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:11.082924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1013/2038] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:10 2022] POST /identity/v3/projects => generated 383 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:11.084646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '415b09885347444481719e72ecb1475a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:11.085522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '415b09885347444481719e72ecb1475a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:11.085790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:11.086416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '415b09885347444481719e72ecb1475a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:11.087552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:11.096382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cfab77aa-af09-4b48-8397-e6b46bd03ed7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '415b09885347444481719e72ecb1475a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:11.130641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1026/2039] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:10 2022] DELETE /identity/v3/users/415b09885347444481719e72ecb1475a => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:11.135269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ad719-5edf-4a9c-b347-cf164b2022d8 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.135701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ad719-5edf-4a9c-b347-cf164b2022d8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.135979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ad719-5edf-4a9c-b347-cf164b2022d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.142659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:11.163033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-205ad719-5edf-4a9c-b347-cf164b2022d8 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:11.164248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-205ad719-5edf-4a9c-b347-cf164b2022d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:11.192256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:11.274444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1027/2040] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:11.281984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205ad719-5edf-4a9c-b347-cf164b2022d8 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:11.283503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-205ad719-5edf-4a9c-b347-cf164b2022d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:11.305235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2863cec5-8e26-4736-b8d6-5aad72ae0d78 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:11.327170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f4879-febe-45cb-b7f2-266d17c7a146 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:11.327814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f4879-febe-45cb-b7f2-266d17c7a146 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.328101 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f4879-febe-45cb-b7f2-266d17c7a146 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.328380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f4879-febe-45cb-b7f2-266d17c7a146 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.339336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2f4879-febe-45cb-b7f2-266d17c7a146 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:11.346760 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2f4879-febe-45cb-b7f2-266d17c7a146 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:11.401086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205ad719-5edf-4a9c-b347-cf164b2022d8 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:11.410388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1014/2041] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:11 2022] POST /identity/v3/users => generated 292 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:11.412719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cc9b57-fce8-4867-9f51-7452f6d43d81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:11.414056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cc9b57-fce8-4867-9f51-7452f6d43d81 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.414344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cc9b57-fce8-4867-9f51-7452f6d43d81 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.414618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cc9b57-fce8-4867-9f51-7452f6d43d81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.420258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f4879-febe-45cb-b7f2-266d17c7a146 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:11.425177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cc9b57-fce8-4867-9f51-7452f6d43d81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:11.427459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cc9b57-fce8-4867-9f51-7452f6d43d81 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:11.434023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f4879-febe-45cb-b7f2-266d17c7a146 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:11.439356 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235fd2ac-6a27-445a-9534-41bad79bf872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:11.440500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235fd2ac-6a27-445a-9534-41bad79bf872 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.441158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235fd2ac-6a27-445a-9534-41bad79bf872 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.441730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235fd2ac-6a27-445a-9534-41bad79bf872 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.442788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235fd2ac-6a27-445a-9534-41bad79bf872 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:11.446062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235fd2ac-6a27-445a-9534-41bad79bf872 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:11.456312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1028/2042] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:11.465383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1015/2043] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:11 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:11.470597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cc9b57-fce8-4867-9f51-7452f6d43d81 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:11.491052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cc9b57-fce8-4867-9f51-7452f6d43d81 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:11.500416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0609f5-58a9-464c-b3fb-e45c72af3a75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:11.501705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0609f5-58a9-464c-b3fb-e45c72af3a75 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.502217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0609f5-58a9-464c-b3fb-e45c72af3a75 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.502762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0609f5-58a9-464c-b3fb-e45c72af3a75 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.503895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0609f5-58a9-464c-b3fb-e45c72af3a75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:11.507596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0609f5-58a9-464c-b3fb-e45c72af3a75 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:11.530819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1029/2044] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:11 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:11.532516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235fd2ac-6a27-445a-9534-41bad79bf872 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:11.545727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235fd2ac-6a27-445a-9534-41bad79bf872 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:11.552260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:11.553036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.553719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.553719 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] PATH_INFO: `/v3/projects/ba52fe3a7e094de1a8e40d50f726685b/users/860adb29213a4641a28ee7610be6cae2/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.613523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba52fe3a7e094de1a8e40d50f726685b, user_id=860adb29213a4641a28ee7610be6cae2, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:11.616993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:11.643493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1016/2045] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:11 2022] PUT /identity/v3/projects/ba52fe3a7e094de1a8e40d50f726685b/users/860adb29213a4641a28ee7610be6cae2/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:11.655149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae78d9b-8af5-4a91-ac4c-6b05a1531cda None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.655514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae78d9b-8af5-4a91-ac4c-6b05a1531cda None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.655860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae78d9b-8af5-4a91-ac4c-6b05a1531cda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.725679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-cae78d9b-8af5-4a91-ac4c-6b05a1531cda None None] MFA Rules not processed for user `860adb29213a4641a28ee7610be6cae2`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:11.728114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cae78d9b-8af5-4a91-ac4c-6b05a1531cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:11.817855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:11.820230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:11.928158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1030/2046] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:11.931212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae78d9b-8af5-4a91-ac4c-6b05a1531cda None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:11.954458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42574308-bf75-4ec0-9b59-c9f413fbc54f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:11.962100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:11.963826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:11.964304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:11.965304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] PATH_INFO: `/v3/projects/fda8b4f99c564368a9a1f08e8789cfe1` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:11.967610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=fda8b4f99c564368a9a1f08e8789cfe1)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:11.971177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:12.040219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:12.046275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fda8b4f99c564368a9a1f08e8789cfe1', 'request_id': 'req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.059111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fda8b4f99c564368a9a1f08e8789cfe1', 'request_id': 'req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.072634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fda8b4f99c564368a9a1f08e8789cfe1', 'request_id': 'req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.073142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae78d9b-8af5-4a91-ac4c-6b05a1531cda None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:12.080551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:12.093551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] The token cache is being invalidate because project fda8b4f99c564368a9a1f08e8789cfe1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:12.093866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.094842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.097871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1017/2047] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:11 2022] DELETE /identity/v3/projects/fda8b4f99c564368a9a1f08e8789cfe1 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:12.102316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:12.104882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:12.184042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:12.184927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:12.185357 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:12.185776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:12.186596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:12.188824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:12.211036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c3f75a-37bf-4ff6-8e71-d63fb50118bc admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:12.220206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:12.220972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:12.221313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:12.221656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] PATH_INFO: `/v3/users/48aead6c7470406c983f8a65618d1453` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:12.225734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1031/2048] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:11 2022] POST /identity/v3/projects => generated 383 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:12.226463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:12.226777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] RBAC: Authorizing `identity:delete_user(user_id=48aead6c7470406c983f8a65618d1453)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:12.228340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:12.229035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:12.306194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48aead6c7470406c983f8a65618d1453'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.308056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48aead6c7470406c983f8a65618d1453'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.310310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2b2303-c6a1-4ac0-8ff7-ad689decd783 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:12.323826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff060d7-09ef-46af-9943-e18c8d73da46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:12.323826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff060d7-09ef-46af-9943-e18c8d73da46 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:12.323826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff060d7-09ef-46af-9943-e18c8d73da46 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:12.323826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff060d7-09ef-46af-9943-e18c8d73da46 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:12.324830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff060d7-09ef-46af-9943-e18c8d73da46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:12.324830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff060d7-09ef-46af-9943-e18c8d73da46 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:12.340093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48aead6c7470406c983f8a65618d1453'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.352144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1032/2049] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:12 2022] POST /identity/v3/users => generated 292 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:12.352399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48aead6c7470406c983f8a65618d1453'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.353276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff060d7-09ef-46af-9943-e18c8d73da46 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:12.366121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff060d7-09ef-46af-9943-e18c8d73da46 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:12.366800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48aead6c7470406c983f8a65618d1453'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:12.372513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce742ec9-6b6d-4327-ada1-ef103cf55d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:12.373176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce742ec9-6b6d-4327-ada1-ef103cf55d4a admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:12.373454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce742ec9-6b6d-4327-ada1-ef103cf55d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:12.373725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce742ec9-6b6d-4327-ada1-ef103cf55d4a admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:12.374389 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce742ec9-6b6d-4327-ada1-ef103cf55d4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:12.375283 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1018/2050] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:12 2022] DELETE /identity/v3/users/48aead6c7470406c983f8a65618d1453 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:12.379551 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce742ec9-6b6d-4327-ada1-ef103cf55d4a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:12.398039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1033/2051] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:12 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:12.400135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:12.401976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:12.533254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0a1d2-87fc-477d-82e9-302e8f03f00e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:12.549623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29f70f0-26da-4701-bd22-6fe965d1400d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:12.553696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29f70f0-26da-4701-bd22-6fe965d1400d admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:12.555134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29f70f0-26da-4701-bd22-6fe965d1400d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:12.555522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29f70f0-26da-4701-bd22-6fe965d1400d admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:12.556307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29f70f0-26da-4701-bd22-6fe965d1400d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:12.562973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29f70f0-26da-4701-bd22-6fe965d1400d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:12.586603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1019/2052] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:12 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:12.591740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce742ec9-6b6d-4327-ada1-ef103cf55d4a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:12.603962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce742ec9-6b6d-4327-ada1-ef103cf55d4a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:12.609921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5c5fd7-c864-402e-90a6-f537fc37c3fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:12.610725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5c5fd7-c864-402e-90a6-f537fc37c3fb admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:12.611152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5c5fd7-c864-402e-90a6-f537fc37c3fb admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:12.611529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5c5fd7-c864-402e-90a6-f537fc37c3fb admin admin] PATH_INFO: `/v3/projects/59643e8c3565487ab1f591a6953d9eae/users/3693568952754f0aba447ce6175e33c7/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:12.633153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5c5fd7-c864-402e-90a6-f537fc37c3fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=59643e8c3565487ab1f591a6953d9eae, user_id=3693568952754f0aba447ce6175e33c7, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:12.637662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5c5fd7-c864-402e-90a6-f537fc37c3fb admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:12.657253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1034/2053] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:12 2022] PUT /identity/v3/projects/59643e8c3565487ab1f591a6953d9eae/users/3693568952754f0aba447ce6175e33c7/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:12.668686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af188d3a-c6c0-43e8-b26e-f7c136e8fbe0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:12.668979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af188d3a-c6c0-43e8-b26e-f7c136e8fbe0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:12.669535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af188d3a-c6c0-43e8-b26e-f7c136e8fbe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:12.717355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:12.726497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-af188d3a-c6c0-43e8-b26e-f7c136e8fbe0 None None] MFA Rules not processed for user `3693568952754f0aba447ce6175e33c7`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:12.733400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-af188d3a-c6c0-43e8-b26e-f7c136e8fbe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:12.938961 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1020/2054] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:12.945163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5c5fd7-c864-402e-90a6-f537fc37c3fb admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:12.965262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af188d3a-c6c0-43e8-b26e-f7c136e8fbe0 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:12.966979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-af188d3a-c6c0-43e8-b26e-f7c136e8fbe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:13.096932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:13.164856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5c5fd7-c864-402e-90a6-f537fc37c3fb admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:13.174563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011bd92f-18fe-4bb0-bc90-8f063819d07d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:13.175378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd92f-18fe-4bb0-bc90-8f063819d07d admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:13.175665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd92f-18fe-4bb0-bc90-8f063819d07d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:13.175933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd92f-18fe-4bb0-bc90-8f063819d07d admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:13.176771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011bd92f-18fe-4bb0-bc90-8f063819d07d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:13.179687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011bd92f-18fe-4bb0-bc90-8f063819d07d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:13.184745 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af188d3a-c6c0-43e8-b26e-f7c136e8fbe0 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:13.198394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:13.199352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:13.199775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:13.200198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] PATH_INFO: `/v3/projects/1b1ee70b12a7429ca3c3a8103b79454b` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:13.207829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b1ee70b12a7429ca3c3a8103b79454b)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:13.209787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:13.229091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1035/2055] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:12 2022] POST /identity/v3/projects => generated 383 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:13.233341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011bd92f-18fe-4bb0-bc90-8f063819d07d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:13.319769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b1ee70b12a7429ca3c3a8103b79454b', 'request_id': 'req-25e8da7e-9376-4ffd-92f8-8c08dd06942c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.328193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b1ee70b12a7429ca3c3a8103b79454b', 'request_id': 'req-25e8da7e-9376-4ffd-92f8-8c08dd06942c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.337988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b1ee70b12a7429ca3c3a8103b79454b', 'request_id': 'req-25e8da7e-9376-4ffd-92f8-8c08dd06942c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.352140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] The token cache is being invalidate because project 1b1ee70b12a7429ca3c3a8103b79454b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:13.352751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.354142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.360314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1021/2056] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:12 2022] DELETE /identity/v3/projects/1b1ee70b12a7429ca3c3a8103b79454b => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:13.362448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:13.365275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:13.374006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011bd92f-18fe-4bb0-bc90-8f063819d07d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:13.431778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:13.473701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:13.474336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:13.475146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:13.475432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:13.476314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:13.479751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:13.483863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e8da7e-9376-4ffd-92f8-8c08dd06942c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:13.491058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:13.492031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:13.492516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:13.493084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] PATH_INFO: `/v3/users/b51e1d0ae1fc432ea563c84554fa2efd` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:13.495444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b51e1d0ae1fc432ea563c84554fa2efd)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:13.499850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:13.521410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1036/2057] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:13 2022] POST /identity/v3/users => generated 298 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:13.526254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:13.532567 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:13.562489 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18b8463-4283-44bd-bf45-252c0cea4876 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:13.572571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a5947e-b3c7-4edb-9316-00cae6405e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:13.573217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a5947e-b3c7-4edb-9316-00cae6405e7c admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:13.574140 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a5947e-b3c7-4edb-9316-00cae6405e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:13.574252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a5947e-b3c7-4edb-9316-00cae6405e7c admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:13.576581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a5947e-b3c7-4edb-9316-00cae6405e7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:13.581056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a5947e-b3c7-4edb-9316-00cae6405e7c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:13.587084 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b51e1d0ae1fc432ea563c84554fa2efd'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.588422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b51e1d0ae1fc432ea563c84554fa2efd'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.598992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b51e1d0ae1fc432ea563c84554fa2efd'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.606878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1037/2058] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:13 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:13.609017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a5947e-b3c7-4edb-9316-00cae6405e7c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:13.623077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b51e1d0ae1fc432ea563c84554fa2efd'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.637188 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a5947e-b3c7-4edb-9316-00cae6405e7c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:13.643303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1815d98-582e-48a3-83fb-859d0e10c4c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:13.643985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1815d98-582e-48a3-83fb-859d0e10c4c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:13.644267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1815d98-582e-48a3-83fb-859d0e10c4c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:13.644541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1815d98-582e-48a3-83fb-859d0e10c4c8 admin admin] PATH_INFO: `/v3/projects/f9c6ed7fd15d46ceaf45c31ff5431d64/users/03b7dfceaf1d4ce09df59976f9a9d93b/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:13.644693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b51e1d0ae1fc432ea563c84554fa2efd'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:13.653443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1022/2059] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:13 2022] DELETE /identity/v3/users/b51e1d0ae1fc432ea563c84554fa2efd => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:13.708544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1815d98-582e-48a3-83fb-859d0e10c4c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9c6ed7fd15d46ceaf45c31ff5431d64, user_id=03b7dfceaf1d4ce09df59976f9a9d93b, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:13.712902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1815d98-582e-48a3-83fb-859d0e10c4c8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:13.744356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1038/2060] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:13 2022] PUT /identity/v3/projects/f9c6ed7fd15d46ceaf45c31ff5431d64/users/03b7dfceaf1d4ce09df59976f9a9d93b/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:13.747369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:13.749035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:13.904371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c1a78e-2d26-479b-a595-dd8a10dc0cd2 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:14.007608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:14.015935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187b328e-3a50-45a1-93e4-303a4cd702d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:14.017032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b328e-3a50-45a1-93e4-303a4cd702d4 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:14.018399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b328e-3a50-45a1-93e4-303a4cd702d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:14.018685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b328e-3a50-45a1-93e4-303a4cd702d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:14.019758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187b328e-3a50-45a1-93e4-303a4cd702d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:14.021637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187b328e-3a50-45a1-93e4-303a4cd702d4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:14.034925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1023/2061] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:13 2022] GET /identity/v3/roles => generated 1420 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:14.038991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1815d98-582e-48a3-83fb-859d0e10c4c8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:14.055962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1815d98-582e-48a3-83fb-859d0e10c4c8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:14.061178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:14.062105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:14.062550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:14.063185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] PATH_INFO: `/v3/projects/f9c6ed7fd15d46ceaf45c31ff5431d64/users/03b7dfceaf1d4ce09df59976f9a9d93b/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:14.065714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9c6ed7fd15d46ceaf45c31ff5431d64, user_id=03b7dfceaf1d4ce09df59976f9a9d93b, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:14.069130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:14.090181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1039/2062] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:14 2022] PUT /identity/v3/projects/f9c6ed7fd15d46ceaf45c31ff5431d64/users/03b7dfceaf1d4ce09df59976f9a9d93b/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:14.103576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5557a6-2f47-420f-b873-43a8d89aa4d5 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:14.103888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5557a6-2f47-420f-b873-43a8d89aa4d5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:14.104170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5557a6-2f47-420f-b873-43a8d89aa4d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:14.156671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-ca5557a6-2f47-420f-b873-43a8d89aa4d5 None None] MFA Rules not processed for user `03b7dfceaf1d4ce09df59976f9a9d93b`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:14.158368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ca5557a6-2f47-420f-b873-43a8d89aa4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:14.300975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:14.303176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:14.328453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:14.353833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1024/2063] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:14 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:14.363592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5557a6-2f47-420f-b873-43a8d89aa4d5 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:14.368916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ca5557a6-2f47-420f-b873-43a8d89aa4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:14.472574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e41c53-29bc-41fa-bd84-86263f891017 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:14.492732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:14.493625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:14.499358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:14.499866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] PATH_INFO: `/v3/projects/257061481c1f4bd1a2fc29ab228ebd08` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:14.510591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] RBAC: Authorizing `identity:delete_project(project_id=257061481c1f4bd1a2fc29ab228ebd08)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:14.513870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:14.547998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5557a6-2f47-420f-b873-43a8d89aa4d5 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:14.552405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2808cea-d6a3-477b-bdf5-4f32f5f1281d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:14.553056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2808cea-d6a3-477b-bdf5-4f32f5f1281d service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:14.553426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2808cea-d6a3-477b-bdf5-4f32f5f1281d service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:14.553756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2808cea-d6a3-477b-bdf5-4f32f5f1281d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:14.557568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d2808cea-d6a3-477b-bdf5-4f32f5f1281d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:14.635295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '257061481c1f4bd1a2fc29ab228ebd08', 'request_id': 'req-cb02d530-7ecc-46b9-aa1c-5504a072a103'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:14.655664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '257061481c1f4bd1a2fc29ab228ebd08', 'request_id': 'req-cb02d530-7ecc-46b9-aa1c-5504a072a103'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:14.666528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '257061481c1f4bd1a2fc29ab228ebd08', 'request_id': 'req-cb02d530-7ecc-46b9-aa1c-5504a072a103'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:14.692158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] The token cache is being invalidate because project 257061481c1f4bd1a2fc29ab228ebd08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:14.692689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:14.693636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb02d530-7ecc-46b9-aa1c-5504a072a103 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:14.695667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1040/2064] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:14 2022] DELETE /identity/v3/projects/257061481c1f4bd1a2fc29ab228ebd08 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:14.702768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa58cc85-0f9b-4cff-acb0-591a6c4a643b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:14.703496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa58cc85-0f9b-4cff-acb0-591a6c4a643b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:14.703629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa58cc85-0f9b-4cff-acb0-591a6c4a643b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:14.720429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2808cea-d6a3-477b-bdf5-4f32f5f1281d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:14.722759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2808cea-d6a3-477b-bdf5-4f32f5f1281d service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:14.748778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-fa58cc85-0f9b-4cff-acb0-591a6c4a643b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:14.753085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1025/2065] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:14 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:14.753680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-fa58cc85-0f9b-4cff-acb0-591a6c4a643b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:14.880793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1041/2066] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:14.891423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42282ced-dbfc-41c1-89c9-e34633172898 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:14.891720 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42282ced-dbfc-41c1-89c9-e34633172898 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:14.891996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42282ced-dbfc-41c1-89c9-e34633172898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:14.925967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-42282ced-dbfc-41c1-89c9-e34633172898 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:14.928139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-42282ced-dbfc-41c1-89c9-e34633172898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:14.956481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1026/2067] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:14.956875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa58cc85-0f9b-4cff-acb0-591a6c4a643b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:14.977181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa58cc85-0f9b-4cff-acb0-591a6c4a643b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:14.983323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:14.984082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:14.984442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:14.985081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:14.986879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:14.987367 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:14.989528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.014789 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1042/2068] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:42:15.018881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42282ced-dbfc-41c1-89c9-e34633172898 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.046091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42282ced-dbfc-41c1-89c9-e34633172898 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.054851 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4020675-c1e6-4969-b0b6-f8cf3ae5d52a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.054851 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4020675-c1e6-4969-b0b6-f8cf3ae5d52a admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.055188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4020675-c1e6-4969-b0b6-f8cf3ae5d52a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.055481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4020675-c1e6-4969-b0b6-f8cf3ae5d52a admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.056244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4020675-c1e6-4969-b0b6-f8cf3ae5d52a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:15.059427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4020675-c1e6-4969-b0b6-f8cf3ae5d52a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.120166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1027/2069] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:15 2022] POST /identity/v3/projects => generated 377 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:15.121275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.138854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff2c1e2-98a5-484e-8715-246d8f52cf58 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.186901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:15.277885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21a3e99-4df1-4e1d-9756-4e203ae98865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.279544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a3e99-4df1-4e1d-9756-4e203ae98865 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.280025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a3e99-4df1-4e1d-9756-4e203ae98865 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.280516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21a3e99-4df1-4e1d-9756-4e203ae98865 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.281458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21a3e99-4df1-4e1d-9756-4e203ae98865 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:15.296283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21a3e99-4df1-4e1d-9756-4e203ae98865 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.361529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1043/2070] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:15 2022] POST /identity/v3/users => generated 289 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:15.365543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4020675-c1e6-4969-b0b6-f8cf3ae5d52a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.374869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:15.383285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4020675-c1e6-4969-b0b6-f8cf3ae5d52a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.389697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.391119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.391119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.391322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.392122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:15.394122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.413193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1028/2071] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:15 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:15.417163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21a3e99-4df1-4e1d-9756-4e203ae98865 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.432129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21a3e99-4df1-4e1d-9756-4e203ae98865 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.440561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504555b-2b57-4458-ae0e-a529b80467ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.440561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504555b-2b57-4458-ae0e-a529b80467ae admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.440561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504555b-2b57-4458-ae0e-a529b80467ae admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.440561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504555b-2b57-4458-ae0e-a529b80467ae admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.440941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0504555b-2b57-4458-ae0e-a529b80467ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:15.441631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.441934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0504555b-2b57-4458-ae0e-a529b80467ae admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.443563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:15.459020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1044/2072] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:15 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:15.466130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504555b-2b57-4458-ae0e-a529b80467ae admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.466748 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b987ac46-63ce-48a6-a46a-21bfa760c06e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.472097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.472906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.473426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.473701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] PATH_INFO: `/v3/users/72c45a0b5705455c8bb3f09053e325cf` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.475100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] RBAC: Authorizing `identity:delete_user(user_id=72c45a0b5705455c8bb3f09053e325cf)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:15.481291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.483068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504555b-2b57-4458-ae0e-a529b80467ae admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.493215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.493215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.493215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.493617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] PATH_INFO: `/v3/projects/385669c5693b403aaa26f64deb5e3c1e/users/46b30f43a3834147bf2df80c179d1bb2/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.528027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] RBAC: Authorizing `identity:create_grant(project_id=385669c5693b403aaa26f64deb5e3c1e, user_id=46b30f43a3834147bf2df80c179d1bb2, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:15.531261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.548946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1045/2073] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:15 2022] PUT /identity/v3/projects/385669c5693b403aaa26f64deb5e3c1e/users/46b30f43a3834147bf2df80c179d1bb2/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:15.549803 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.551520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:15.567221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72c45a0b5705455c8bb3f09053e325cf'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.568104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72c45a0b5705455c8bb3f09053e325cf'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.580299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72c45a0b5705455c8bb3f09053e325cf'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.589369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72c45a0b5705455c8bb3f09053e325cf'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.604298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-634eade5-99c9-48f6-b8a5-32e2501d790d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72c45a0b5705455c8bb3f09053e325cf'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.610814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1029/2074] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:15 2022] DELETE /identity/v3/users/72c45a0b5705455c8bb3f09053e325cf => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:15.612072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c749c6b8-312b-4d1c-be85-ac65f59eacb6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.612353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c749c6b8-312b-4d1c-be85-ac65f59eacb6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.612625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c749c6b8-312b-4d1c-be85-ac65f59eacb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.650789 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eba4df5-f1e0-4b12-bdae-0aa48cbbb750 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.651338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c749c6b8-312b-4d1c-be85-ac65f59eacb6 None None] MFA Rules not processed for user `46b30f43a3834147bf2df80c179d1bb2`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:15.653264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c749c6b8-312b-4d1c-be85-ac65f59eacb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:15.740723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.741450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.741732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.742061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] PATH_INFO: `/v3/users/80f895cc698644b189c81240b9192823` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.743382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] RBAC: Authorizing `identity:delete_user(user_id=80f895cc698644b189c81240b9192823)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:15.746292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.764549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1030/2075] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:15.765347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c749c6b8-312b-4d1c-be85-ac65f59eacb6 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.767113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c749c6b8-312b-4d1c-be85-ac65f59eacb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:15.783372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c749c6b8-312b-4d1c-be85-ac65f59eacb6 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.799572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:15.800968 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.801665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.802022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.802367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.803708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:15.806298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:15.842744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80f895cc698644b189c81240b9192823'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.852987 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80f895cc698644b189c81240b9192823'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.853714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80f895cc698644b189c81240b9192823'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.854407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80f895cc698644b189c81240b9192823'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.854666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1031/2076] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:15 2022] POST /identity/v3/projects => generated 381 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:15.862882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.867572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:15.869465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80f895cc698644b189c81240b9192823'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:15.905707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1046/2077] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:15 2022] DELETE /identity/v3/users/80f895cc698644b189c81240b9192823 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:15.907096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:15.908783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:15.917133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:15.954186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10293925-6d33-412e-a86a-2c5ce53dccfc admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.990333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6194d27-7a39-4939-a483-3876cdc8f16b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:15.995477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:15.996647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:15.996967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:15.997207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] PATH_INFO: `/v3/projects/5b5cee4eb87f4428aa0c68502f2eaab1` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:15.998391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b5cee4eb87f4428aa0c68502f2eaab1)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.000375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.028378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba35caf-f06b-420c-b077-1964ede1d96b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.029003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba35caf-f06b-420c-b077-1964ede1d96b admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.029312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba35caf-f06b-420c-b077-1964ede1d96b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.029569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba35caf-f06b-420c-b077-1964ede1d96b admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.030196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba35caf-f06b-420c-b077-1964ede1d96b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.032276 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba35caf-f06b-420c-b077-1964ede1d96b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.050275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b5cee4eb87f4428aa0c68502f2eaab1', 'request_id': 'req-7b009357-928c-4df4-a5ed-9f572792167f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:16.054201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1032/2078] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:15 2022] POST /identity/v3/users => generated 297 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:16.055689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910e358-2b7a-46ce-b2f1-10227b0fe110 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.055995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910e358-2b7a-46ce-b2f1-10227b0fe110 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.056260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910e358-2b7a-46ce-b2f1-10227b0fe110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.056681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b5cee4eb87f4428aa0c68502f2eaab1', 'request_id': 'req-7b009357-928c-4df4-a5ed-9f572792167f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:16.063395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b5cee4eb87f4428aa0c68502f2eaab1', 'request_id': 'req-7b009357-928c-4df4-a5ed-9f572792167f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:16.077311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1910e358-2b7a-46ce-b2f1-10227b0fe110 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:16.077705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] The token cache is being invalidate because project 5b5cee4eb87f4428aa0c68502f2eaab1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:16.078006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:16.078538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1910e358-2b7a-46ce-b2f1-10227b0fe110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.078694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:16.080210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1047/2079] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:15 2022] DELETE /identity/v3/projects/5b5cee4eb87f4428aa0c68502f2eaab1 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:16.083563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.085239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.130263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:16.146681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1033/2080] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:16.149067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808441e8-d9b6-4d20-9786-f3b3163a4c1c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.149558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808441e8-d9b6-4d20-9786-f3b3163a4c1c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.150065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808441e8-d9b6-4d20-9786-f3b3163a4c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.157709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b009357-928c-4df4-a5ed-9f572792167f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.163508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.164476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.165062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.168761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.168761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.169991 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.175460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-808441e8-d9b6-4d20-9786-f3b3163a4c1c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:16.177429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-808441e8-d9b6-4d20-9786-f3b3163a4c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.181580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1048/2081] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:16 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:16.182404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.185055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.188635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1034/2082] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:16.189605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808441e8-d9b6-4d20-9786-f3b3163a4c1c None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.199985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808441e8-d9b6-4d20-9786-f3b3163a4c1c None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.204953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fed6a5-fa69-4c4b-a038-630bec0b395c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.205601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fed6a5-fa69-4c4b-a038-630bec0b395c admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.205879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fed6a5-fa69-4c4b-a038-630bec0b395c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.206150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fed6a5-fa69-4c4b-a038-630bec0b395c admin admin] PATH_INFO: `/v3/projects/fa7bbc1b7cc14597b71a91f6ddb220ab/users/c81a044721694ba88f3d509a517488ee/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.221873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fed6a5-fa69-4c4b-a038-630bec0b395c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa7bbc1b7cc14597b71a91f6ddb220ab, user_id=c81a044721694ba88f3d509a517488ee, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.224552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fed6a5-fa69-4c4b-a038-630bec0b395c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.243140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1035/2083] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:16 2022] PUT /identity/v3/projects/fa7bbc1b7cc14597b71a91f6ddb220ab/users/c81a044721694ba88f3d509a517488ee/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:16.249025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c977f55-0272-4069-9a96-9b39f78005d6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.249025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c977f55-0272-4069-9a96-9b39f78005d6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.249025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c977f55-0272-4069-9a96-9b39f78005d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.255914 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0bf9b9-b764-4f00-877f-f4ba465e9f6c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.267995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5c977f55-0272-4069-9a96-9b39f78005d6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:16.269417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5c977f55-0272-4069-9a96-9b39f78005d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.312738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:16.321587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a210353-3ff3-4200-bc1f-b968ffebabf0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.321894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a210353-3ff3-4200-bc1f-b968ffebabf0 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.322054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a210353-3ff3-4200-bc1f-b968ffebabf0 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.322379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a210353-3ff3-4200-bc1f-b968ffebabf0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.359829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1036/2084] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:16.359829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c977f55-0272-4069-9a96-9b39f78005d6 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.382478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c977f55-0272-4069-9a96-9b39f78005d6 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.388456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c277b1-8969-4ba4-b42b-bea340ac15b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.389399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c277b1-8969-4ba4-b42b-bea340ac15b2 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.389399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c277b1-8969-4ba4-b42b-bea340ac15b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.392147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c277b1-8969-4ba4-b42b-bea340ac15b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.392147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c277b1-8969-4ba4-b42b-bea340ac15b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.392147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c277b1-8969-4ba4-b42b-bea340ac15b2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.410457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1037/2085] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:16 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:16.410457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c277b1-8969-4ba4-b42b-bea340ac15b2 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.414007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a210353-3ff3-4200-bc1f-b968ffebabf0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.416315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a210353-3ff3-4200-bc1f-b968ffebabf0 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.422249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c277b1-8969-4ba4-b42b-bea340ac15b2 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.431745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1049/2086] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:16.431837 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.431958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a210353-3ff3-4200-bc1f-b968ffebabf0 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.432036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.432036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.432036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.438908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:16.439827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.441935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.442917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a210353-3ff3-4200-bc1f-b968ffebabf0 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.456844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba270160-2315-4012-985b-4f12e9c85b38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.457538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba270160-2315-4012-985b-4f12e9c85b38 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.457819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba270160-2315-4012-985b-4f12e9c85b38 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.458092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba270160-2315-4012-985b-4f12e9c85b38 admin admin] PATH_INFO: `/v3/projects/fa7bbc1b7cc14597b71a91f6ddb220ab/users/c81a044721694ba88f3d509a517488ee/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.460716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1038/2087] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:42:16.461219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba270160-2315-4012-985b-4f12e9c85b38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa7bbc1b7cc14597b71a91f6ddb220ab, user_id=c81a044721694ba88f3d509a517488ee, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.464174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba270160-2315-4012-985b-4f12e9c85b38 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.465706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.477517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854c0cf-e65b-4a2c-a242-17729c5c5d52 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.479110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1050/2088] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:16 2022] PUT /identity/v3/projects/fa7bbc1b7cc14597b71a91f6ddb220ab/users/c81a044721694ba88f3d509a517488ee/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:16.490939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fb456d-9c68-4c34-99f0-e7b3f8f4d67a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.491224 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fb456d-9c68-4c34-99f0-e7b3f8f4d67a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.491506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fb456d-9c68-4c34-99f0-e7b3f8f4d67a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.530468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-59fb456d-9c68-4c34-99f0-e7b3f8f4d67a None None] MFA Rules not processed for user `c81a044721694ba88f3d509a517488ee`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:16.531853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-59fb456d-9c68-4c34-99f0-e7b3f8f4d67a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.556988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:16.622415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.623334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.623651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.625061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.625061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.627143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.666999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1039/2089] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:16 2022] POST /identity/v3/projects => generated 393 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:16.668995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.671005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.681778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1051/2090] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:16 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:16.681778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fb456d-9c68-4c34-99f0-e7b3f8f4d67a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.696166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fb456d-9c68-4c34-99f0-e7b3f8f4d67a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.777612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763944e-a066-40c6-90bd-e78988aa823f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.783164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.783948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.784640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.784928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.787389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.795394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.796608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.796608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.796608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.797260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.799714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.832380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1052/2091] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:16 2022] POST /identity/v3/users => generated 297 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:16.833911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.838103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.854422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5bfbed-da15-40c9-a254-d2886334f108 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.859804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f71f395-dc80-46bf-971a-0491257deab5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.860432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f71f395-dc80-46bf-971a-0491257deab5 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.860712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f71f395-dc80-46bf-971a-0491257deab5 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.860998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f71f395-dc80-46bf-971a-0491257deab5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.876321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:16.881540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.883990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.904643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1040/2092] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:16.905908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.920198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994cfa94-a95b-47ad-b3a6-097f9d7539c7 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.933078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:16.934712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:16.935126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:16.935409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:16.936140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.937925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:16.950721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1041/2093] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:16 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:16.951585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:16.963447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:16.994550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f71f395-dc80-46bf-971a-0491257deab5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:16.995610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d986100e-ae4d-4014-b210-89ae07be05bf admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:16.997872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f71f395-dc80-46bf-971a-0491257deab5 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.003922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:17.004545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.004826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.005403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] PATH_INFO: `/v3/projects/0da1a62f992b4d3f86adaa2d0230d35b` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.007071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0da1a62f992b4d3f86adaa2d0230d35b)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:17.010295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.012930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1053/2094] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:16 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:17.013865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f71f395-dc80-46bf-971a-0491257deab5 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.027596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f71f395-dc80-46bf-971a-0491257deab5 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:17.034334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c42ed-fa89-4941-9522-c460bfd238a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:17.035588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c42ed-fa89-4941-9522-c460bfd238a0 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.036310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c42ed-fa89-4941-9522-c460bfd238a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.036707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c42ed-fa89-4941-9522-c460bfd238a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.037496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0c42ed-fa89-4941-9522-c460bfd238a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:17.039905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0c42ed-fa89-4941-9522-c460bfd238a0 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.058957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1054/2095] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:17 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:17.066090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c42ed-fa89-4941-9522-c460bfd238a0 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.083958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c42ed-fa89-4941-9522-c460bfd238a0 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:17.094691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:17.095511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.096249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.096581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] PATH_INFO: `/v3/projects/8ad722485ac34d1796f03003fbcde07a/users/8b46b3fa1df24c80ab7315c4ff44c064/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.112153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0da1a62f992b4d3f86adaa2d0230d35b', 'request_id': 'req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.124167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0da1a62f992b4d3f86adaa2d0230d35b', 'request_id': 'req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.131931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ad722485ac34d1796f03003fbcde07a, user_id=8b46b3fa1df24c80ab7315c4ff44c064, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:17.135023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.135871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0da1a62f992b4d3f86adaa2d0230d35b', 'request_id': 'req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.153484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:17.153693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] The token cache is being invalidate because project 0da1a62f992b4d3f86adaa2d0230d35b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:17.153786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.154596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.156489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1042/2096] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:16 2022] DELETE /identity/v3/projects/0da1a62f992b4d3f86adaa2d0230d35b => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:17.157417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.160004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:17.162268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1055/2097] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:17 2022] PUT /identity/v3/projects/8ad722485ac34d1796f03003fbcde07a/users/8b46b3fa1df24c80ab7315c4ff44c064/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:17.163277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.166363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:17.266842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42781d45-1618-45b7-bed1-deaa02875d1e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:17.273484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:17.274563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.274981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.275259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] PATH_INFO: `/v3/users/bb0944cbbc9f4cbf8a1859e39d858fd2` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.276934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb0944cbbc9f4cbf8a1859e39d858fd2)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:17.278802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27e584e-6e49-4ac5-9b7e-e3fb98100cb9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:17.281372 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.288123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:17.289610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.290075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.290656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] PATH_INFO: `/v3/users/2aa0cdf3ca35427ab0cf4401afc6e65e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.292254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2aa0cdf3ca35427ab0cf4401afc6e65e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:17.295594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.373563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2aa0cdf3ca35427ab0cf4401afc6e65e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.374310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2aa0cdf3ca35427ab0cf4401afc6e65e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.383865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb0944cbbc9f4cbf8a1859e39d858fd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.390336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2aa0cdf3ca35427ab0cf4401afc6e65e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.399499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb0944cbbc9f4cbf8a1859e39d858fd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.400522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb0944cbbc9f4cbf8a1859e39d858fd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.403537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2aa0cdf3ca35427ab0cf4401afc6e65e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.404484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb0944cbbc9f4cbf8a1859e39d858fd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.418288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d4095080-a28d-4fbb-9b30-69448261a9b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb0944cbbc9f4cbf8a1859e39d858fd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.420225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:17.434022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c755ec53-28c7-4a83-912e-44acc3a5802d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2aa0cdf3ca35427ab0cf4401afc6e65e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.446557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1043/2098] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:17 2022] DELETE /identity/v3/users/2aa0cdf3ca35427ab0cf4401afc6e65e => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:17.448372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c7fe81-d4b0-4dba-9b72-ac54033ce55d None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.448876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c7fe81-d4b0-4dba-9b72-ac54033ce55d None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.449539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c7fe81-d4b0-4dba-9b72-ac54033ce55d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.451800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1056/2099] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:17 2022] DELETE /identity/v3/users/bb0944cbbc9f4cbf8a1859e39d858fd2 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:17.453470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645790e-b45a-4845-9b2c-f6e76f113599 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.453829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645790e-b45a-4845-9b2c-f6e76f113599 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.454182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645790e-b45a-4845-9b2c-f6e76f113599 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.482443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5645790e-b45a-4845-9b2c-f6e76f113599 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:17.488236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d6c7fe81-d4b0-4dba-9b72-ac54033ce55d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:17.488499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5645790e-b45a-4845-9b2c-f6e76f113599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:17.490676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d6c7fe81-d4b0-4dba-9b72-ac54033ce55d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:17.569595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1057/2100] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:17.580797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645790e-b45a-4845-9b2c-f6e76f113599 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.582560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5645790e-b45a-4845-9b2c-f6e76f113599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:17.622666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:17.644723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1044/2101] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:17.644723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c7fe81-d4b0-4dba-9b72-ac54033ce55d None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.652853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d6c7fe81-d4b0-4dba-9b72-ac54033ce55d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:17.671944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c7fe81-d4b0-4dba-9b72-ac54033ce55d None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:17.682260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:17.683658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.684008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.684698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] PATH_INFO: `/v3/users/46b30f43a3834147bf2df80c179d1bb2` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.686008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46b30f43a3834147bf2df80c179d1bb2)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:17.689213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.696798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645790e-b45a-4845-9b2c-f6e76f113599 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:17.702023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3635d677-0fd7-4c69-a2ec-70486ee0f4c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:17.703692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3635d677-0fd7-4c69-a2ec-70486ee0f4c9 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.704328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3635d677-0fd7-4c69-a2ec-70486ee0f4c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.704837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3635d677-0fd7-4c69-a2ec-70486ee0f4c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.724543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3635d677-0fd7-4c69-a2ec-70486ee0f4c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:17.726371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3635d677-0fd7-4c69-a2ec-70486ee0f4c9 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.744794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1058/2102] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:17.747616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3635d677-0fd7-4c69-a2ec-70486ee0f4c9 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.767899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3635d677-0fd7-4c69-a2ec-70486ee0f4c9 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:17.776789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46b30f43a3834147bf2df80c179d1bb2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.777724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46b30f43a3834147bf2df80c179d1bb2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.793699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:17.794333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.796920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.797155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.797753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46b30f43a3834147bf2df80c179d1bb2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.815645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46b30f43a3834147bf2df80c179d1bb2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.822187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:17.826237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:17.837986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e48fd1ca-d608-432c-b113-424f000d4c20 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46b30f43a3834147bf2df80c179d1bb2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:17.842323 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1059/2103] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:17.844155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.846232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:17.856440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1045/2104] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:17 2022] DELETE /identity/v3/users/46b30f43a3834147bf2df80c179d1bb2 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:17.859940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac90e3d-51fb-4f82-99b2-f9a77a465ed5 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:17.860713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac90e3d-51fb-4f82-99b2-f9a77a465ed5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:17.861082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac90e3d-51fb-4f82-99b2-f9a77a465ed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:17.894257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5ac90e3d-51fb-4f82-99b2-f9a77a465ed5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:17.895517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5ac90e3d-51fb-4f82-99b2-f9a77a465ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:17.940966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a2de26-e447-46bc-8beb-cfb2ec0a01ac service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:17.981329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:17.994694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1046/2105] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:17.995398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac90e3d-51fb-4f82-99b2-f9a77a465ed5 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:17.997416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5ac90e3d-51fb-4f82-99b2-f9a77a465ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:18.010470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:18.011142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.012047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.012341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.015318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:18.031468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac90e3d-51fb-4f82-99b2-f9a77a465ed5 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:18.039736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:18.040916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.041605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.042046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] PATH_INFO: `/v3/projects/6fc71ea0219443bca187d837a00e443b` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.044274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] RBAC: Authorizing `identity:delete_project(project_id=6fc71ea0219443bca187d837a00e443b)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:18.047380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:18.109140 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:18.111625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:18.113527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6fc71ea0219443bca187d837a00e443b', 'request_id': 'req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.122849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6fc71ea0219443bca187d837a00e443b', 'request_id': 'req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.125830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1060/2106] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:17 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:18.136100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6fc71ea0219443bca187d837a00e443b', 'request_id': 'req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.138808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:18.142781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:18.159375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] The token cache is being invalidate because project 6fc71ea0219443bca187d837a00e443b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:18.159706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.160747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5aec9a19-1ee0-4ce0-a985-c6cb22c0ecfc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.166423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1047/2107] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:17 2022] DELETE /identity/v3/projects/6fc71ea0219443bca187d837a00e443b => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:18.167975 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a0d9a-8852-4145-a3d4-2beba3318d9f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.168489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a0d9a-8852-4145-a3d4-2beba3318d9f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.169072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a0d9a-8852-4145-a3d4-2beba3318d9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.185264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:18.203533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-397a0d9a-8852-4145-a3d4-2beba3318d9f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:18.205152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-397a0d9a-8852-4145-a3d4-2beba3318d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:18.262403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09688b7c-8af9-49fe-a804-5f8f6d8e72d1 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:18.304476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1048/2108] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:18.305346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397a0d9a-8852-4145-a3d4-2beba3318d9f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:18.306623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-397a0d9a-8852-4145-a3d4-2beba3318d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:18.330467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397a0d9a-8852-4145-a3d4-2beba3318d9f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:18.335765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:18.336393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.336680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.336948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] PATH_INFO: `/v3/projects/f3b80a1e2fcc414bb50798dfd6571d99` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.338704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3b80a1e2fcc414bb50798dfd6571d99)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:18.341319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:18.343513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a97c132-b5f7-48aa-aa3f-30490beec445 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:18.344551 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97c132-b5f7-48aa-aa3f-30490beec445 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.344895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97c132-b5f7-48aa-aa3f-30490beec445 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.345110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97c132-b5f7-48aa-aa3f-30490beec445 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.347882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1a97c132-b5f7-48aa-aa3f-30490beec445 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:18.367480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a97c132-b5f7-48aa-aa3f-30490beec445 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:18.369992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a97c132-b5f7-48aa-aa3f-30490beec445 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:18.384251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1061/2109] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:18.385634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f4a81-d493-4c37-82bc-82463e1a5869 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.386072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f4a81-d493-4c37-82bc-82463e1a5869 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.387167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999f4a81-d493-4c37-82bc-82463e1a5869 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.404962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3b80a1e2fcc414bb50798dfd6571d99', 'request_id': 'req-b4745411-d2c3-454c-a57c-900c5578f7ec'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.413234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3b80a1e2fcc414bb50798dfd6571d99', 'request_id': 'req-b4745411-d2c3-454c-a57c-900c5578f7ec'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.421604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3b80a1e2fcc414bb50798dfd6571d99', 'request_id': 'req-b4745411-d2c3-454c-a57c-900c5578f7ec'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.434398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-999f4a81-d493-4c37-82bc-82463e1a5869 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:18.435710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] The token cache is being invalidate because project f3b80a1e2fcc414bb50798dfd6571d99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:18.436005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.436386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-999f4a81-d493-4c37-82bc-82463e1a5869 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:18.436696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.440126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1049/2110] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:18 2022] DELETE /identity/v3/projects/f3b80a1e2fcc414bb50798dfd6571d99 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:18.603299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1062/2111] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:18.607353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:18.663807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4745411-d2c3-454c-a57c-900c5578f7ec admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:18.678175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:18.679504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.679971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.680510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.681721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:18.682352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:18.685554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:18.688420 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:18.713404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1050/2112] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:42:18.717485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999f4a81-d493-4c37-82bc-82463e1a5869 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:18.728676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999f4a81-d493-4c37-82bc-82463e1a5869 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:18.737818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68195e39-693f-4cf4-aaee-6bf0d74f9c0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:18.738664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68195e39-693f-4cf4-aaee-6bf0d74f9c0d admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.739001 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68195e39-693f-4cf4-aaee-6bf0d74f9c0d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.739521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68195e39-693f-4cf4-aaee-6bf0d74f9c0d admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.740716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68195e39-693f-4cf4-aaee-6bf0d74f9c0d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:18.742844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68195e39-693f-4cf4-aaee-6bf0d74f9c0d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:18.761016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:18.763599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:18.787372 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1063/2113] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:18 2022] POST /identity/v3/projects => generated 385 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:18.788920 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beeb7cf-a8c7-4383-a5c0-06781927b590 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:18.791100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68195e39-693f-4cf4-aaee-6bf0d74f9c0d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:18.868264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:18.874095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:18.875066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.875437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.876808 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] PATH_INFO: `/v3/projects/385669c5693b403aaa26f64deb5e3c1e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.879346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=385669c5693b403aaa26f64deb5e3c1e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:18.881618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:18.882440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68195e39-693f-4cf4-aaee-6bf0d74f9c0d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:18.891976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db75745e-2438-48d4-9361-caec7ca1875d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:18.892610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db75745e-2438-48d4-9361-caec7ca1875d admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:18.893029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db75745e-2438-48d4-9361-caec7ca1875d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:18.893322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db75745e-2438-48d4-9361-caec7ca1875d admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:18.893962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db75745e-2438-48d4-9361-caec7ca1875d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:18.896265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db75745e-2438-48d4-9361-caec7ca1875d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:18.922235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1064/2114] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:18 2022] POST /identity/v3/users => generated 293 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:18.926453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db75745e-2438-48d4-9361-caec7ca1875d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:18.937240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db75745e-2438-48d4-9361-caec7ca1875d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:18.937738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '385669c5693b403aaa26f64deb5e3c1e', 'request_id': 'req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.944582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '385669c5693b403aaa26f64deb5e3c1e', 'request_id': 'req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.954025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '385669c5693b403aaa26f64deb5e3c1e', 'request_id': 'req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.969677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] The token cache is being invalidate because project 385669c5693b403aaa26f64deb5e3c1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:18.970124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.972377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:18.974697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1051/2115] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:18 2022] DELETE /identity/v3/projects/385669c5693b403aaa26f64deb5e3c1e => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:18.978639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:18.980665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:19.017658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.019328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.020329 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.020895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.021917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.024707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.039087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1065/2116] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:18 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:19.043778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:19.045742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:19.055131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe12e5b-f2b6-4865-bbe1-a71da219d1b2 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:19.061210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e3ce4b-d834-4e75-8acd-a04eedae9b4a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:19.061400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.061876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.062441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.062670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] PATH_INFO: `/v3/users/c81a044721694ba88f3d509a517488ee` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.063999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=c81a044721694ba88f3d509a517488ee)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.066060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.066352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac858bf-4cdf-42bf-aa89-2c45a53d89f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.067003 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac858bf-4cdf-42bf-aa89-2c45a53d89f5 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.067281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac858bf-4cdf-42bf-aa89-2c45a53d89f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.067580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac858bf-4cdf-42bf-aa89-2c45a53d89f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.068355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac858bf-4cdf-42bf-aa89-2c45a53d89f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.069835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac858bf-4cdf-42bf-aa89-2c45a53d89f5 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.088984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1066/2117] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:19 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:19.093429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac858bf-4cdf-42bf-aa89-2c45a53d89f5 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:19.106956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac858bf-4cdf-42bf-aa89-2c45a53d89f5 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:19.113761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:19.115755 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6795f40-3de2-4bed-90cb-6267306f13e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.117617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6795f40-3de2-4bed-90cb-6267306f13e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.118642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6795f40-3de2-4bed-90cb-6267306f13e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.119377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6795f40-3de2-4bed-90cb-6267306f13e9 admin admin] PATH_INFO: `/v3/projects/93f7ac3224cf42929135e5af81989ecf/users/28273f910cc54510bb72ef750ed4bea8/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.145148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c81a044721694ba88f3d509a517488ee'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:19.146201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c81a044721694ba88f3d509a517488ee'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:19.147078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6795f40-3de2-4bed-90cb-6267306f13e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93f7ac3224cf42929135e5af81989ecf, user_id=28273f910cc54510bb72ef750ed4bea8, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.149968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6795f40-3de2-4bed-90cb-6267306f13e9 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.157148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c81a044721694ba88f3d509a517488ee'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:19.171812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c81a044721694ba88f3d509a517488ee'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:19.175864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1067/2118] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:19 2022] PUT /identity/v3/projects/93f7ac3224cf42929135e5af81989ecf/users/28273f910cc54510bb72ef750ed4bea8/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:19.188910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2abbdc4-553c-4081-b78e-af23c48fd666 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.189220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2abbdc4-553c-4081-b78e-af23c48fd666 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.189595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2abbdc4-553c-4081-b78e-af23c48fd666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.191893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c81a044721694ba88f3d509a517488ee'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:19.210341 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1052/2119] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:18 2022] DELETE /identity/v3/users/c81a044721694ba88f3d509a517488ee => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:19.231169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f2abbdc4-553c-4081-b78e-af23c48fd666 None None] MFA Rules not processed for user `28273f910cc54510bb72ef750ed4bea8`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:19.237239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f2abbdc4-553c-4081-b78e-af23c48fd666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:19.361492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1068/2120] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:19 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:19.366068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:19.367516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:19.531724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9822f1b-b667-4ff6-a0dd-32b0a7a525cd admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:19.538570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.539526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.539824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.540357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.541114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.544400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.596579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2abbdc4-553c-4081-b78e-af23c48fd666 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:19.611362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1053/2121] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:19 2022] POST /identity/v3/projects => generated 387 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:19.617103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2abbdc4-553c-4081-b78e-af23c48fd666 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:19.661015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:19.663655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:19.702843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:19.775840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71d16e2-8a25-48f8-be24-cd6b05db229a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.777393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d16e2-8a25-48f8-be24-cd6b05db229a admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.777393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d16e2-8a25-48f8-be24-cd6b05db229a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.777682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d16e2-8a25-48f8-be24-cd6b05db229a admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.778559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71d16e2-8a25-48f8-be24-cd6b05db229a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.783122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71d16e2-8a25-48f8-be24-cd6b05db229a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.847895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1069/2122] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:19 2022] POST /identity/v3/users => generated 300 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:19.852742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71d16e2-8a25-48f8-be24-cd6b05db229a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:19.873281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71d16e2-8a25-48f8-be24-cd6b05db229a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:19.878641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26133ac-0c06-4b01-8e3d-03b2a75d421d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.879442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26133ac-0c06-4b01-8e3d-03b2a75d421d admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.880898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26133ac-0c06-4b01-8e3d-03b2a75d421d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.881282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26133ac-0c06-4b01-8e3d-03b2a75d421d admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.882198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26133ac-0c06-4b01-8e3d-03b2a75d421d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.884936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26133ac-0c06-4b01-8e3d-03b2a75d421d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.893966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3db5e-6c41-4c24-9d06-7ac892ee756f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:19.905141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1070/2123] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:19 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:19.906939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26133ac-0c06-4b01-8e3d-03b2a75d421d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:19.910381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.911107 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.911468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.911658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] PATH_INFO: `/v3/projects/fa7bbc1b7cc14597b71a91f6ddb220ab` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.916019 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa7bbc1b7cc14597b71a91f6ddb220ab)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.918768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.934027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26133ac-0c06-4b01-8e3d-03b2a75d421d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:19.938205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1230dea-5e97-4156-b6d0-f7d2281bb7d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:19.938979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1230dea-5e97-4156-b6d0-f7d2281bb7d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:19.939268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1230dea-5e97-4156-b6d0-f7d2281bb7d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:19.939549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1230dea-5e97-4156-b6d0-f7d2281bb7d1 admin admin] PATH_INFO: `/v3/projects/63f069590e3b49c6a5021b98faf513f6/users/a83b0b614029424f995db6dcb97071e6/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:19.961971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1230dea-5e97-4156-b6d0-f7d2281bb7d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63f069590e3b49c6a5021b98faf513f6, user_id=a83b0b614029424f995db6dcb97071e6, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:19.966766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1230dea-5e97-4156-b6d0-f7d2281bb7d1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:19.990397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1071/2124] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:19 2022] PUT /identity/v3/projects/63f069590e3b49c6a5021b98faf513f6/users/a83b0b614029424f995db6dcb97071e6/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:19.994458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1230dea-5e97-4156-b6d0-f7d2281bb7d1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.007884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=8,keystone:SELECT=90,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:20.043114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa7bbc1b7cc14597b71a91f6ddb220ab', 'request_id': 'req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:20.050209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa7bbc1b7cc14597b71a91f6ddb220ab', 'request_id': 'req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:20.060775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa7bbc1b7cc14597b71a91f6ddb220ab', 'request_id': 'req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:20.087552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] The token cache is being invalidate because project fa7bbc1b7cc14597b71a91f6ddb220ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:20.087856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:20.090943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb407dd4-e029-4cc2-ad15-cc10e9e4b443 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:20.105238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1054/2125] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:19 2022] DELETE /identity/v3/projects/fa7bbc1b7cc14597b71a91f6ddb220ab => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:20.108331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85244d9d-21b9-4002-896f-66b06d1c5425 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.108615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85244d9d-21b9-4002-896f-66b06d1c5425 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.108899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85244d9d-21b9-4002-896f-66b06d1c5425 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.141463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1230dea-5e97-4156-b6d0-f7d2281bb7d1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.141761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-85244d9d-21b9-4002-896f-66b06d1c5425 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:20.143047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-85244d9d-21b9-4002-896f-66b06d1c5425 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:20.216681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.216966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.217403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.217802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.219871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.226473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1055/2126] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:20.226633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.228063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310375c2-11ab-4451-b288-f5d64c64dcc6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.228351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310375c2-11ab-4451-b288-f5d64c64dcc6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.228625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310375c2-11ab-4451-b288-f5d64c64dcc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.233241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1072/2127] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:19 2022] GET /identity/v3/roles => generated 1420 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:20.238370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.240720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:20.251483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-310375c2-11ab-4451-b288-f5d64c64dcc6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:20.252867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-310375c2-11ab-4451-b288-f5d64c64dcc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:20.257023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbac74a1-f1b4-47e1-91a6-6d64733b6e56 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.263366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623a02f6-1e73-4299-8314-1b2df12996eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.266872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1056/2128] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:20.266961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a02f6-1e73-4299-8314-1b2df12996eb admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.266961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a02f6-1e73-4299-8314-1b2df12996eb admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.266961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a02f6-1e73-4299-8314-1b2df12996eb admin admin] PATH_INFO: `/v3/projects/63f069590e3b49c6a5021b98faf513f6/users/a83b0b614029424f995db6dcb97071e6/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.266961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623a02f6-1e73-4299-8314-1b2df12996eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=63f069590e3b49c6a5021b98faf513f6, user_id=a83b0b614029424f995db6dcb97071e6, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.267973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310375c2-11ab-4451-b288-f5d64c64dcc6 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.270029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623a02f6-1e73-4299-8314-1b2df12996eb admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.277814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310375c2-11ab-4451-b288-f5d64c64dcc6 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.287908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1073/2129] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:20 2022] PUT /identity/v3/projects/63f069590e3b49c6a5021b98faf513f6/users/a83b0b614029424f995db6dcb97071e6/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:20.303522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314db0f6-2e97-4ae3-957d-768770a20998 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.303819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314db0f6-2e97-4ae3-957d-768770a20998 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.304099 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314db0f6-2e97-4ae3-957d-768770a20998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.361814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-314db0f6-2e97-4ae3-957d-768770a20998 None None] MFA Rules not processed for user `a83b0b614029424f995db6dcb97071e6`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:20.363119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-314db0f6-2e97-4ae3-957d-768770a20998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:20.372595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:20.392299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.394121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.394604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.395433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.399344 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:20.399756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.401703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.415378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1057/2130] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:42:20.420386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.437202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade08f55-551f-42c8-9bcb-bf76352a8c2e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.443196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc07bef-af50-4036-8a5e-06e8935f7a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.444288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc07bef-af50-4036-8a5e-06e8935f7a49 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.444671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc07bef-af50-4036-8a5e-06e8935f7a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.445047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc07bef-af50-4036-8a5e-06e8935f7a49 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.445811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc07bef-af50-4036-8a5e-06e8935f7a49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.448869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc07bef-af50-4036-8a5e-06e8935f7a49 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.479210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1074/2131] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:20 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:20.490690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314db0f6-2e97-4ae3-957d-768770a20998 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.492897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-314db0f6-2e97-4ae3-957d-768770a20998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:20.497936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1058/2132] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:20 2022] POST /identity/v3/projects => generated 375 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:20.501898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc07bef-af50-4036-8a5e-06e8935f7a49 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.553842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:20.582326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314db0f6-2e97-4ae3-957d-768770a20998 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.618284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc07bef-af50-4036-8a5e-06e8935f7a49 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.625144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b49199a-4737-420e-918c-8536d92ea015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.625999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49199a-4737-420e-918c-8536d92ea015 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.626281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49199a-4737-420e-918c-8536d92ea015 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.626555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49199a-4737-420e-918c-8536d92ea015 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.627246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b49199a-4737-420e-918c-8536d92ea015 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.630480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b49199a-4737-420e-918c-8536d92ea015 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.657732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a579af3-e47c-4eca-a5c9-de8ddf53eb72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.658361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a579af3-e47c-4eca-a5c9-de8ddf53eb72 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.658646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a579af3-e47c-4eca-a5c9-de8ddf53eb72 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.658954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a579af3-e47c-4eca-a5c9-de8ddf53eb72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.662214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1059/2133] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:20 2022] POST /identity/v3/users => generated 288 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:20.663340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b49199a-4737-420e-918c-8536d92ea015 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.675895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b49199a-4737-420e-918c-8536d92ea015 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.681848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddccde52-f1a1-463e-af9f-7db0adf7a1ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.682777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddccde52-f1a1-463e-af9f-7db0adf7a1ed admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.683855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddccde52-f1a1-463e-af9f-7db0adf7a1ed admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.684233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddccde52-f1a1-463e-af9f-7db0adf7a1ed admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.685006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddccde52-f1a1-463e-af9f-7db0adf7a1ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.687631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddccde52-f1a1-463e-af9f-7db0adf7a1ed admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.705298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1060/2134] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:20 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:20.706507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddccde52-f1a1-463e-af9f-7db0adf7a1ed admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.716830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddccde52-f1a1-463e-af9f-7db0adf7a1ed admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.722138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48e697a-cc5e-450e-97f3-6305240a0ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.722669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48e697a-cc5e-450e-97f3-6305240a0ac5 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.722995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48e697a-cc5e-450e-97f3-6305240a0ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.723423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48e697a-cc5e-450e-97f3-6305240a0ac5 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.724076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48e697a-cc5e-450e-97f3-6305240a0ac5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.726044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48e697a-cc5e-450e-97f3-6305240a0ac5 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.740123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1061/2135] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:20 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:20.744725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48e697a-cc5e-450e-97f3-6305240a0ac5 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:20.749203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a579af3-e47c-4eca-a5c9-de8ddf53eb72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.751144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a579af3-e47c-4eca-a5c9-de8ddf53eb72 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.756383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48e697a-cc5e-450e-97f3-6305240a0ac5 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:20.761319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8797d5a-81fd-4701-955d-430b14b4bf59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:20.762122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8797d5a-81fd-4701-955d-430b14b4bf59 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.762486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8797d5a-81fd-4701-955d-430b14b4bf59 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.762891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8797d5a-81fd-4701-955d-430b14b4bf59 admin admin] PATH_INFO: `/v3/projects/86c655d027dd4b59a6752e5abe9e2bf0/users/8fe5792e5fc849cf853a600426e41079/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.764998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1075/2136] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:20 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:20.779987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8797d5a-81fd-4701-955d-430b14b4bf59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=86c655d027dd4b59a6752e5abe9e2bf0, user_id=8fe5792e5fc849cf853a600426e41079, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:20.783067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8797d5a-81fd-4701-955d-430b14b4bf59 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:20.821772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9f281b-2a78-4b8b-89eb-b43dca3f7b38 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:20.822335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9f281b-2a78-4b8b-89eb-b43dca3f7b38 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:20.824415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9f281b-2a78-4b8b-89eb-b43dca3f7b38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:20.827260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1062/2137] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:20 2022] PUT /identity/v3/projects/86c655d027dd4b59a6752e5abe9e2bf0/users/8fe5792e5fc849cf853a600426e41079/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:20.857780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:20.888331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-db9f281b-2a78-4b8b-89eb-b43dca3f7b38 None None] MFA Rules not processed for user `8fe5792e5fc849cf853a600426e41079`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:20.893791 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-db9f281b-2a78-4b8b-89eb-b43dca3f7b38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:21.028216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1076/2138] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:20 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:21.043895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8797d5a-81fd-4701-955d-430b14b4bf59 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:21.044105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9f281b-2a78-4b8b-89eb-b43dca3f7b38 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:21.131578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9f281b-2a78-4b8b-89eb-b43dca3f7b38 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:21.145975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f9a2f1-8109-46ab-93b1-e575b243a9d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:21.146793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9a2f1-8109-46ab-93b1-e575b243a9d6 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.147861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9a2f1-8109-46ab-93b1-e575b243a9d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.149792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9a2f1-8109-46ab-93b1-e575b243a9d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.151981 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f9a2f1-8109-46ab-93b1-e575b243a9d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:21.155826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f9a2f1-8109-46ab-93b1-e575b243a9d6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:21.209933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8797d5a-81fd-4701-955d-430b14b4bf59 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:21.226252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1077/2139] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:21 2022] POST /identity/v3/projects => generated 377 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:21.230746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f9a2f1-8109-46ab-93b1-e575b243a9d6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:21.233336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:21.234174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.234602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.235038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] PATH_INFO: `/v3/users/28273f910cc54510bb72ef750ed4bea8` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.236513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28273f910cc54510bb72ef750ed4bea8)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:21.238632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:21.334357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28273f910cc54510bb72ef750ed4bea8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:21.335677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28273f910cc54510bb72ef750ed4bea8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:21.348709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28273f910cc54510bb72ef750ed4bea8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:21.359680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28273f910cc54510bb72ef750ed4bea8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:21.368123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=86,keystone:INSERT=5,keystone:DELETE=9 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:21.379285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c25fe316-9027-4653-9fc1-2ad41e802117 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28273f910cc54510bb72ef750ed4bea8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:21.389817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1063/2140] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:21 2022] DELETE /identity/v3/users/28273f910cc54510bb72ef750ed4bea8 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:21.398617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fd09ac-4900-494a-b2d0-a3658596684a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.399030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fd09ac-4900-494a-b2d0-a3658596684a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.399442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fd09ac-4900-494a-b2d0-a3658596684a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.410781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f9a2f1-8109-46ab-93b1-e575b243a9d6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:21.436307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-82fd09ac-4900-494a-b2d0-a3658596684a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:21.437675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-82fd09ac-4900-494a-b2d0-a3658596684a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:21.486936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:21.562072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:21.563256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.563718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.564059 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.564830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:21.567247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:21.595396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1064/2141] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:21.597992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1078/2142] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:21 2022] POST /identity/v3/users => generated 289 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:21.603059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fd09ac-4900-494a-b2d0-a3658596684a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:21.603771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:21.606158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:21.607251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-82fd09ac-4900-494a-b2d0-a3658596684a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:21.621560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fd09ac-4900-494a-b2d0-a3658596684a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:21.628248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcac985-48f0-4633-a937-eb739ca764c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:21.629250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcac985-48f0-4633-a937-eb739ca764c0 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.629738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcac985-48f0-4633-a937-eb739ca764c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.630928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcac985-48f0-4633-a937-eb739ca764c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.631931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcac985-48f0-4633-a937-eb739ca764c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:21.634009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcac985-48f0-4633-a937-eb739ca764c0 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:21.647349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1065/2143] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:21 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:21.650977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcac985-48f0-4633-a937-eb739ca764c0 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:21.663797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcac985-48f0-4633-a937-eb739ca764c0 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:21.671007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc73e1-4fbc-4ef8-b4d2-d5a32903d51e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:21.672994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc73e1-4fbc-4ef8-b4d2-d5a32903d51e admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.673676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc73e1-4fbc-4ef8-b4d2-d5a32903d51e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.674628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc73e1-4fbc-4ef8-b4d2-d5a32903d51e admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.675570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dc73e1-4fbc-4ef8-b4d2-d5a32903d51e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:21.677860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dc73e1-4fbc-4ef8-b4d2-d5a32903d51e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:21.697768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1066/2144] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:21 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:21.701971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7bcffd-99fb-4f0d-9c43-28240afa8a0d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:21.703175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc73e1-4fbc-4ef8-b4d2-d5a32903d51e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:21.711250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e103e1-2b2e-4b47-9285-1e343b38a11f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:21.716550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e103e1-2b2e-4b47-9285-1e343b38a11f service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.717234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e103e1-2b2e-4b47-9285-1e343b38a11f service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.717577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e103e1-2b2e-4b47-9285-1e343b38a11f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.717889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc73e1-4fbc-4ef8-b4d2-d5a32903d51e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:21.731795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf6aae-d978-4eeb-a61d-ec849c2d0e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:21.734337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf6aae-d978-4eeb-a61d-ec849c2d0e76 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.734626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf6aae-d978-4eeb-a61d-ec849c2d0e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.734929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf6aae-d978-4eeb-a61d-ec849c2d0e76 admin admin] PATH_INFO: `/v3/projects/ffe4fcc4bba142ef9e608278fdae7443/users/05a8e834732f4c35aa567adddaa11447/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.740297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e103e1-2b2e-4b47-9285-1e343b38a11f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:21.742150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e103e1-2b2e-4b47-9285-1e343b38a11f service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:21.757007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1079/2145] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:21.777676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadf6aae-d978-4eeb-a61d-ec849c2d0e76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffe4fcc4bba142ef9e608278fdae7443, user_id=05a8e834732f4c35aa567adddaa11447, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:21.782297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadf6aae-d978-4eeb-a61d-ec849c2d0e76 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:21.822066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1067/2146] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:21 2022] PUT /identity/v3/projects/ffe4fcc4bba142ef9e608278fdae7443/users/05a8e834732f4c35aa567adddaa11447/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:21.835274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4264fbc7-0124-437f-a3ac-1d5ffe59b90e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:21.835998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4264fbc7-0124-437f-a3ac-1d5ffe59b90e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:21.836461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4264fbc7-0124-437f-a3ac-1d5ffe59b90e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:21.884167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-4264fbc7-0124-437f-a3ac-1d5ffe59b90e None None] MFA Rules not processed for user `05a8e834732f4c35aa567adddaa11447`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:21.888762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4264fbc7-0124-437f-a3ac-1d5ffe59b90e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:22.009619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1080/2147] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:22.014459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf6aae-d978-4eeb-a61d-ec849c2d0e76 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.130836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf6aae-d978-4eeb-a61d-ec849c2d0e76 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.137618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:22.138345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.138697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.139083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:22.139799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:22.142815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:22.183717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1068/2148] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:22 2022] POST /identity/v3/projects => generated 377 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:22.185368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4264fbc7-0124-437f-a3ac-1d5ffe59b90e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.215988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4264fbc7-0124-437f-a3ac-1d5ffe59b90e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.225323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.227151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:22.283375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:22.296881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7de4ee4-a9cf-47e1-9578-648292338090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:22.297796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7de4ee4-a9cf-47e1-9578-648292338090 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.298143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7de4ee4-a9cf-47e1-9578-648292338090 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.298485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7de4ee4-a9cf-47e1-9578-648292338090 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:22.299276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7de4ee4-a9cf-47e1-9578-648292338090 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:22.301359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7de4ee4-a9cf-47e1-9578-648292338090 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:22.307087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:DELETE=1,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:22.334775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1081/2149] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:22 2022] POST /identity/v3/users => generated 295 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:22.338722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7de4ee4-a9cf-47e1-9578-648292338090 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.354472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7de4ee4-a9cf-47e1-9578-648292338090 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.363496 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ae3a8-279f-4cb3-bebc-db4037e452a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:22.365225 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ae3a8-279f-4cb3-bebc-db4037e452a3 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.365793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ae3a8-279f-4cb3-bebc-db4037e452a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.366473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ae3a8-279f-4cb3-bebc-db4037e452a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:22.368241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82ae3a8-279f-4cb3-bebc-db4037e452a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:22.370222 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82ae3a8-279f-4cb3-bebc-db4037e452a3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:22.383996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1082/2150] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:22 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:22.384398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4529465-86c3-4366-94de-5fe869309449 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.388518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ae3a8-279f-4cb3-bebc-db4037e452a3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.390363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:22.391204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.391602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.391896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] PATH_INFO: `/v3/projects/93f7ac3224cf42929135e5af81989ecf` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:22.393136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] RBAC: Authorizing `identity:delete_project(project_id=93f7ac3224cf42929135e5af81989ecf)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:22.395517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:22.406002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ae3a8-279f-4cb3-bebc-db4037e452a3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.421350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9959ca29-0bc7-4d3f-a55c-6686f4ab5304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:22.422314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9959ca29-0bc7-4d3f-a55c-6686f4ab5304 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.422769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9959ca29-0bc7-4d3f-a55c-6686f4ab5304 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.423349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9959ca29-0bc7-4d3f-a55c-6686f4ab5304 admin admin] PATH_INFO: `/v3/projects/1530ebce582d4378ae92165ca3ba4121/users/b64422f1c0864c6ebfe89a839b46e948/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:22.465388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9959ca29-0bc7-4d3f-a55c-6686f4ab5304 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1530ebce582d4378ae92165ca3ba4121, user_id=b64422f1c0864c6ebfe89a839b46e948, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:22.472604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9959ca29-0bc7-4d3f-a55c-6686f4ab5304 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:22.491425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93f7ac3224cf42929135e5af81989ecf', 'request_id': 'req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.500353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1083/2151] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:22 2022] PUT /identity/v3/projects/1530ebce582d4378ae92165ca3ba4121/users/b64422f1c0864c6ebfe89a839b46e948/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:22.506295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9959ca29-0bc7-4d3f-a55c-6686f4ab5304 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.506874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93f7ac3224cf42929135e5af81989ecf', 'request_id': 'req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.528195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93f7ac3224cf42929135e5af81989ecf', 'request_id': 'req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.546694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] The token cache is being invalidate because project 93f7ac3224cf42929135e5af81989ecf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:22.547259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.550480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.556151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1069/2152] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:22 2022] DELETE /identity/v3/projects/93f7ac3224cf42929135e5af81989ecf => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:22.566465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.568879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:22.670606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9959ca29-0bc7-4d3f-a55c-6686f4ab5304 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.687657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:22.688590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.688946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.689398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:22.690259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:22.692285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:22.709931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1084/2153] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:22 2022] GET /identity/v3/roles => generated 1420 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:22.711395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.718340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:22.751957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e769deeb-27dd-4f39-b03c-fdc319f9e200 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.754256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda38fbe-5bf3-4c71-9c94-fe3a1687a8db admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.759827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:22.760261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:22.760744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.762468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.762468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.762674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.762782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] PATH_INFO: `/v3/projects/1530ebce582d4378ae92165ca3ba4121/users/b64422f1c0864c6ebfe89a839b46e948/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:22.762873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] PATH_INFO: `/v3/users/a83b0b614029424f995db6dcb97071e6` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:22.764831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a83b0b614029424f995db6dcb97071e6)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:22.764961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1530ebce582d4378ae92165ca3ba4121, user_id=b64422f1c0864c6ebfe89a839b46e948, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:22.767723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:22.768150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:22.795631 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1085/2154] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:22 2022] PUT /identity/v3/projects/1530ebce582d4378ae92165ca3ba4121/users/b64422f1c0864c6ebfe89a839b46e948/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:22.796645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:22.807117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:22.877326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a83b0b614029424f995db6dcb97071e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.878151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a83b0b614029424f995db6dcb97071e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.892717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a83b0b614029424f995db6dcb97071e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.895537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:22.905207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a83b0b614029424f995db6dcb97071e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.927353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ac100363-535e-4354-b488-a3e7271bf7c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a83b0b614029424f995db6dcb97071e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:22.945409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2560ce3-47bb-481f-997c-991c2edc1415 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:22.947697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1070/2155] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:22 2022] DELETE /identity/v3/users/a83b0b614029424f995db6dcb97071e6 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:22.949636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8283d7c2-6ccf-435e-9b01-98494e858d57 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:22.949935 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8283d7c2-6ccf-435e-9b01-98494e858d57 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:22.950213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8283d7c2-6ccf-435e-9b01-98494e858d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:23.005522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-8283d7c2-6ccf-435e-9b01-98494e858d57 None None] MFA Rules not processed for user `b64422f1c0864c6ebfe89a839b46e948`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:23.007401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8283d7c2-6ccf-435e-9b01-98494e858d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.024052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:23.065603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:23.066253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:23.066698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:23.067078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] PATH_INFO: `/v3/users/860adb29213a4641a28ee7610be6cae2` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:23.068972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=860adb29213a4641a28ee7610be6cae2)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:23.071570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:23.164124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1071/2156] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:22 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:23.181573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8283d7c2-6ccf-435e-9b01-98494e858d57 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:23.183663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8283d7c2-6ccf-435e-9b01-98494e858d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.199304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '860adb29213a4641a28ee7610be6cae2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.218673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '860adb29213a4641a28ee7610be6cae2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.219753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '860adb29213a4641a28ee7610be6cae2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.220506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '860adb29213a4641a28ee7610be6cae2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.230090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ca0d4fb4-df5e-4829-996a-cb26cea71cbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '860adb29213a4641a28ee7610be6cae2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.274538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1086/2157] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:22 2022] DELETE /identity/v3/users/860adb29213a4641a28ee7610be6cae2 => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:23.276121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e886de4c-a1c3-4374-b4fe-d4ff27374f48 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:23.276415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e886de4c-a1c3-4374-b4fe-d4ff27374f48 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:23.276753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e886de4c-a1c3-4374-b4fe-d4ff27374f48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:23.337491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8283d7c2-6ccf-435e-9b01-98494e858d57 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:23.339453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e886de4c-a1c3-4374-b4fe-d4ff27374f48 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:23.341715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e886de4c-a1c3-4374-b4fe-d4ff27374f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.450294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:23.451240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:23.451240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:23.451480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:23.454764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.492096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1087/2158] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:23.493825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e886de4c-a1c3-4374-b4fe-d4ff27374f48 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:23.498169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e886de4c-a1c3-4374-b4fe-d4ff27374f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.514456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e886de4c-a1c3-4374-b4fe-d4ff27374f48 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:23.529599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:23.530238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:23.530568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:23.531822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] PATH_INFO: `/v3/projects/63f069590e3b49c6a5021b98faf513f6` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:23.533997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=63f069590e3b49c6a5021b98faf513f6)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:23.539798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:23.565060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=3,keystone:SELECT=97 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:23.624373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63f069590e3b49c6a5021b98faf513f6', 'request_id': 'req-39863d66-b842-4da1-81e5-8246a9c675e2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.629431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:23.647507 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63f069590e3b49c6a5021b98faf513f6', 'request_id': 'req-39863d66-b842-4da1-81e5-8246a9c675e2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.648431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:23.650187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:23.664165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63f069590e3b49c6a5021b98faf513f6', 'request_id': 'req-39863d66-b842-4da1-81e5-8246a9c675e2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.674292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1072/2159] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:23.685772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:23.690177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.694318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] The token cache is being invalidate because project 63f069590e3b49c6a5021b98faf513f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:23.694979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.696077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-39863d66-b842-4da1-81e5-8246a9c675e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:23.707816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1088/2160] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:23 2022] DELETE /identity/v3/projects/63f069590e3b49c6a5021b98faf513f6 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:23.712716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a08b68-47d6-4c2d-a407-ec482d2a0730 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:23.713237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a08b68-47d6-4c2d-a407-ec482d2a0730 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:23.714155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a08b68-47d6-4c2d-a407-ec482d2a0730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:23.757474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e4a08b68-47d6-4c2d-a407-ec482d2a0730 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:23.760153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e4a08b68-47d6-4c2d-a407-ec482d2a0730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.827085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510e1b24-eabb-4d83-bf78-85782b8aecaa service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:23.903185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1089/2161] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:23.903185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e700c7-4159-438d-b838-469aa5c96b66 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:23.903185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e700c7-4159-438d-b838-469aa5c96b66 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:23.903185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e700c7-4159-438d-b838-469aa5c96b66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:23.945379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-39e700c7-4159-438d-b838-469aa5c96b66 None None] MFA Rules not processed for user `8b46b3fa1df24c80ab7315c4ff44c064`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:23.947572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-39e700c7-4159-438d-b838-469aa5c96b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.951482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6123bd21-10c6-44d8-aa5b-71a71948bfd4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:23.952324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123bd21-10c6-44d8-aa5b-71a71948bfd4 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:23.952788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123bd21-10c6-44d8-aa5b-71a71948bfd4 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:23.954223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123bd21-10c6-44d8-aa5b-71a71948bfd4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:23.957270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6123bd21-10c6-44d8-aa5b-71a71948bfd4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:23.977872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6123bd21-10c6-44d8-aa5b-71a71948bfd4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:23.980807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6123bd21-10c6-44d8-aa5b-71a71948bfd4 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:23.992177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1073/2162] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:23.994722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c86d8ba-266b-4065-aa92-427684e118dc None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:23.996320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c86d8ba-266b-4065-aa92-427684e118dc None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:23.996645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c86d8ba-266b-4065-aa92-427684e118dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.029295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-8c86d8ba-266b-4065-aa92-427684e118dc None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:24.031034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8c86d8ba-266b-4065-aa92-427684e118dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:24.040700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1074/2163] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:24.045162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c86d8ba-266b-4065-aa92-427684e118dc None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.056091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1090/2164] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:23 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:24.064578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c86d8ba-266b-4065-aa92-427684e118dc None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.068422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4f8e6-2ebe-4fe0-a597-2c18cf72ecf3 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.068769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4f8e6-2ebe-4fe0-a597-2c18cf72ecf3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.069050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4f8e6-2ebe-4fe0-a597-2c18cf72ecf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.078908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.079579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.079869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.080368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.081043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:24.081351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.083241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.100536 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1075/2165] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:42:24.105303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.113619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-47e4f8e6-2ebe-4fe0-a597-2c18cf72ecf3 None None] MFA Rules not processed for user `8b46b3fa1df24c80ab7315c4ff44c064`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:24.116475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-47e4f8e6-2ebe-4fe0-a597-2c18cf72ecf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:24.116812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e52a72-73df-4f4d-b6ae-4fe88673b25e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.127869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7289c54-7080-46ad-81cd-5eebb958bb65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.128602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7289c54-7080-46ad-81cd-5eebb958bb65 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.128810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7289c54-7080-46ad-81cd-5eebb958bb65 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.129024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7289c54-7080-46ad-81cd-5eebb958bb65 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.129799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1091/2166] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:24 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:24.129996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7289c54-7080-46ad-81cd-5eebb958bb65 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.138043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7289c54-7080-46ad-81cd-5eebb958bb65 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.143289 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4f8e6-2ebe-4fe0-a597-2c18cf72ecf3 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.144783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-47e4f8e6-2ebe-4fe0-a597-2c18cf72ecf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:24.185533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1076/2167] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:24 2022] POST /identity/v3/projects => generated 393 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:24.190872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7289c54-7080-46ad-81cd-5eebb958bb65 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.320158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:24.323332 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:24.329990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4f8e6-2ebe-4fe0-a597-2c18cf72ecf3 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.350438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7289c54-7080-46ad-81cd-5eebb958bb65 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.359079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd4b92a-d5cc-498e-b55b-4f971f10262c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.359615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4b92a-d5cc-498e-b55b-4f971f10262c admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.360500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4b92a-d5cc-498e-b55b-4f971f10262c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.360781 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4b92a-d5cc-498e-b55b-4f971f10262c admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.361524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd4b92a-d5cc-498e-b55b-4f971f10262c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.363525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd4b92a-d5cc-498e-b55b-4f971f10262c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.399440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1077/2168] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:24 2022] POST /identity/v3/users => generated 297 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:24.401321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd4b92a-d5cc-498e-b55b-4f971f10262c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.422683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd4b92a-d5cc-498e-b55b-4f971f10262c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.432443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8298508a-2e29-4815-a074-a9951f6611d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.434067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8298508a-2e29-4815-a074-a9951f6611d3 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.434651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8298508a-2e29-4815-a074-a9951f6611d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.435212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8298508a-2e29-4815-a074-a9951f6611d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.436449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8298508a-2e29-4815-a074-a9951f6611d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.439894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8298508a-2e29-4815-a074-a9951f6611d3 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.442391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8b2b6b-c0d4-48c4-8bc3-4f8c46919497 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.443414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8b2b6b-c0d4-48c4-8bc3-4f8c46919497 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.444870 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8b2b6b-c0d4-48c4-8bc3-4f8c46919497 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.446100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8b2b6b-c0d4-48c4-8bc3-4f8c46919497 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.455189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1078/2169] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:24 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:24.459804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8298508a-2e29-4815-a074-a9951f6611d3 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.473368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8298508a-2e29-4815-a074-a9951f6611d3 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.479407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.480915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.481569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.481962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.482817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.484831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.498668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1079/2170] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:24 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:24.502793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.504608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:24.521617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6619437-e9c8-4fc1-b2f5-1d60c9426847 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.527815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.529089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.529390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.529672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] PATH_INFO: `/v3/projects/ba52fe3a7e094de1a8e40d50f726685b` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.531317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba52fe3a7e094de1a8e40d50f726685b)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.533655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.534577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8b2b6b-c0d4-48c4-8bc3-4f8c46919497 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.536239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8b2b6b-c0d4-48c4-8bc3-4f8c46919497 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.548476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1092/2171] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:24 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:24.549326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8b2b6b-c0d4-48c4-8bc3-4f8c46919497 service nova] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.562720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8b2b6b-c0d4-48c4-8bc3-4f8c46919497 service nova] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.569362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3d1a43-985c-489a-bc40-0ef7e1f2bf77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.570155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3d1a43-985c-489a-bc40-0ef7e1f2bf77 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.570536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3d1a43-985c-489a-bc40-0ef7e1f2bf77 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.570850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3d1a43-985c-489a-bc40-0ef7e1f2bf77 admin admin] PATH_INFO: `/v3/projects/bdb8eca30d6849ccb89066cea399b74a/users/d6dd5a84af3e4e22b80b5a068cd2cca3/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.607238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3d1a43-985c-489a-bc40-0ef7e1f2bf77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdb8eca30d6849ccb89066cea399b74a, user_id=d6dd5a84af3e4e22b80b5a068cd2cca3, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.610110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba52fe3a7e094de1a8e40d50f726685b', 'request_id': 'req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.610398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3d1a43-985c-489a-bc40-0ef7e1f2bf77 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.619053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba52fe3a7e094de1a8e40d50f726685b', 'request_id': 'req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.628890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba52fe3a7e094de1a8e40d50f726685b', 'request_id': 'req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.630147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1093/2172] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:24 2022] PUT /identity/v3/projects/bdb8eca30d6849ccb89066cea399b74a/users/d6dd5a84af3e4e22b80b5a068cd2cca3/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:24.647298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e637e8-59ff-4614-85aa-e7874b35e810 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.647900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e637e8-59ff-4614-85aa-e7874b35e810 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.648428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e637e8-59ff-4614-85aa-e7874b35e810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.650331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] The token cache is being invalidate because project ba52fe3a7e094de1a8e40d50f726685b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:24.650701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.651546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.653427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1080/2173] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:24 2022] DELETE /identity/v3/projects/ba52fe3a7e094de1a8e40d50f726685b => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:24.657488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.658937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:24.705457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c9e637e8-59ff-4614-85aa-e7874b35e810 None None] MFA Rules not processed for user `d6dd5a84af3e4e22b80b5a068cd2cca3`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:24.706710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c9e637e8-59ff-4614-85aa-e7874b35e810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:24.743058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66f0cc-a01b-47a1-a7ae-1b83f686bbca admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.827302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:24.835615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1094/2174] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:24 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:24.840707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e637e8-59ff-4614-85aa-e7874b35e810 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.842824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c9e637e8-59ff-4614-85aa-e7874b35e810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:24.848673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.849422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.849780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.850338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] PATH_INFO: `/v3/users/3693568952754f0aba447ce6175e33c7` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.852351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3693568952754f0aba447ce6175e33c7)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.853172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:24.856574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.858592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e637e8-59ff-4614-85aa-e7874b35e810 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:24.873662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f601fc00-bfff-42b6-bd9d-d8b60a182a75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:24.874393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f601fc00-bfff-42b6-bd9d-d8b60a182a75 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:24.874677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f601fc00-bfff-42b6-bd9d-d8b60a182a75 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:24.876020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f601fc00-bfff-42b6-bd9d-d8b60a182a75 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:24.876691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f601fc00-bfff-42b6-bd9d-d8b60a182a75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:24.878653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f601fc00-bfff-42b6-bd9d-d8b60a182a75 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:24.921802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1095/2175] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:24 2022] POST /identity/v3/projects => generated 391 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:24.928072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f601fc00-bfff-42b6-bd9d-d8b60a182a75 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:24.931689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3693568952754f0aba447ce6175e33c7'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.932446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3693568952754f0aba447ce6175e33c7'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.941357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3693568952754f0aba447ce6175e33c7'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.952944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3693568952754f0aba447ce6175e33c7'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.971946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3693568952754f0aba447ce6175e33c7'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:24.989175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1081/2176] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:24 2022] DELETE /identity/v3/users/3693568952754f0aba447ce6175e33c7 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:25.016855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f601fc00-bfff-42b6-bd9d-d8b60a182a75 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:25.108102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405d8e84-042e-493c-bebe-67c1317ec09b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:25.109580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405d8e84-042e-493c-bebe-67c1317ec09b admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.109775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405d8e84-042e-493c-bebe-67c1317ec09b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.110057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405d8e84-042e-493c-bebe-67c1317ec09b admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.110636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405d8e84-042e-493c-bebe-67c1317ec09b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:25.112940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405d8e84-042e-493c-bebe-67c1317ec09b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:25.158478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1096/2177] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:24 2022] POST /identity/v3/users => generated 302 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:42:25.163694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:25.165344 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:25.202103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa26f65-06a9-40ca-96d8-76eeb5a53893 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:25.207539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a40b11-57bc-4698-a35c-7256692b4bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:25.207884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a40b11-57bc-4698-a35c-7256692b4bea admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.208815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a40b11-57bc-4698-a35c-7256692b4bea admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.209044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a40b11-57bc-4698-a35c-7256692b4bea admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.209777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a40b11-57bc-4698-a35c-7256692b4bea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:25.211815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a40b11-57bc-4698-a35c-7256692b4bea admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:25.234146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405d8e84-042e-493c-bebe-67c1317ec09b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:25.246510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1082/2178] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:25 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:25.248109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405d8e84-042e-493c-bebe-67c1317ec09b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:25.255746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:25.256312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.256825 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.257402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] PATH_INFO: `/v3/projects/e6a8dee9fb0c44efaa3f5932b1d7a00f/users/fbb9d3507a3e43a1a3cd2f2dcb6dea70/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.283715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6a8dee9fb0c44efaa3f5932b1d7a00f, user_id=fbb9d3507a3e43a1a3cd2f2dcb6dea70, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:25.286695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:25.316690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a40b11-57bc-4698-a35c-7256692b4bea admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:25.317675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1097/2179] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:25 2022] PUT /identity/v3/projects/e6a8dee9fb0c44efaa3f5932b1d7a00f/users/fbb9d3507a3e43a1a3cd2f2dcb6dea70/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:25.331602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a40b11-57bc-4698-a35c-7256692b4bea admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:25.445534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:25.448383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:25.476799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:25.477442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.477891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.478203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.479073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:25.481048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:25.492838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1083/2180] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:25 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:25.493709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:25.498710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:25.520497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55ceba8-6b2a-43fa-b7e7-8ef4855fe56f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:25.526137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:25.527793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.528260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.528866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] PATH_INFO: `/v3/projects/59643e8c3565487ab1f591a6953d9eae` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.530347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=59643e8c3565487ab1f591a6953d9eae)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:25.544260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:25.584122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600fe5ce-0b5b-426c-bada-9f878837882a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:25.595967 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb15cf58-d022-4c7b-b945-945afa53b932 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:25.596883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15cf58-d022-4c7b-b945-945afa53b932 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.597762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15cf58-d022-4c7b-b945-945afa53b932 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.598317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15cf58-d022-4c7b-b945-945afa53b932 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.600816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-fb15cf58-d022-4c7b-b945-945afa53b932 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:25.613798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:25.620596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:25.639459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59643e8c3565487ab1f591a6953d9eae', 'request_id': 'req-911ec4e2-4c99-43d6-b242-b5e138220a8e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:25.653617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59643e8c3565487ab1f591a6953d9eae', 'request_id': 'req-911ec4e2-4c99-43d6-b242-b5e138220a8e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:25.664793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59643e8c3565487ab1f591a6953d9eae', 'request_id': 'req-911ec4e2-4c99-43d6-b242-b5e138220a8e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:25.678978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] The token cache is being invalidate because project 59643e8c3565487ab1f591a6953d9eae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:25.679351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:25.680041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:25.681757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1084/2181] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:25 2022] DELETE /identity/v3/projects/59643e8c3565487ab1f591a6953d9eae => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:25.682467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:25.684119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:25.701022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb15cf58-d022-4c7b-b945-945afa53b932 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:25.702502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb15cf58-d022-4c7b-b945-945afa53b932 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:25.713332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1098/2182] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:25 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:25.715215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44c364-6893-4cdc-9b5f-c259f9f6ecbb None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.715506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44c364-6893-4cdc-9b5f-c259f9f6ecbb None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.715790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44c364-6893-4cdc-9b5f-c259f9f6ecbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.740952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1b44c364-6893-4cdc-9b5f-c259f9f6ecbb None None] MFA Rules not processed for user `8b46b3fa1df24c80ab7315c4ff44c064`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:25.743665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1b44c364-6893-4cdc-9b5f-c259f9f6ecbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:25.769887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911ec4e2-4c99-43d6-b242-b5e138220a8e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:25.778140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:25.778825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.779099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.779370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] PATH_INFO: `/v3/projects/e6a8dee9fb0c44efaa3f5932b1d7a00f/users/fbb9d3507a3e43a1a3cd2f2dcb6dea70/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.782938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6a8dee9fb0c44efaa3f5932b1d7a00f, user_id=fbb9d3507a3e43a1a3cd2f2dcb6dea70, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:25.785984 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:25.811139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1085/2183] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:25 2022] PUT /identity/v3/projects/e6a8dee9fb0c44efaa3f5932b1d7a00f/users/fbb9d3507a3e43a1a3cd2f2dcb6dea70/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:25.812578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:25.814614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:25.856708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1099/2184] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:25 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:25.856708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7598b2ff-2320-4108-bb10-74b7c534bc55 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.856708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7598b2ff-2320-4108-bb10-74b7c534bc55 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.856708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7598b2ff-2320-4108-bb10-74b7c534bc55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.898304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7598b2ff-2320-4108-bb10-74b7c534bc55 None None] MFA Rules not processed for user `fbb9d3507a3e43a1a3cd2f2dcb6dea70`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:25.900149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7598b2ff-2320-4108-bb10-74b7c534bc55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:25.940932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38cf525-80e8-46f5-964f-6aac06c311dc admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:25.947196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:25.950526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:25.953925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:25.954520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] PATH_INFO: `/v3/users/03b7dfceaf1d4ce09df59976f9a9d93b` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:25.956424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=03b7dfceaf1d4ce09df59976f9a9d93b)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:25.958707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:26.034970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03b7dfceaf1d4ce09df59976f9a9d93b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.035799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03b7dfceaf1d4ce09df59976f9a9d93b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.045976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03b7dfceaf1d4ce09df59976f9a9d93b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.058319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03b7dfceaf1d4ce09df59976f9a9d93b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.070317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:26.081125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03b7dfceaf1d4ce09df59976f9a9d93b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.087590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1100/2185] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:25 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:26.090796 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1086/2186] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:25 2022] DELETE /identity/v3/users/03b7dfceaf1d4ce09df59976f9a9d93b => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:26.106347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7598b2ff-2320-4108-bb10-74b7c534bc55 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:26.112242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7598b2ff-2320-4108-bb10-74b7c534bc55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:26.226679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:26.259198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7598b2ff-2320-4108-bb10-74b7c534bc55 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:26.411554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:26.412788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:26.413090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:26.413421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:26.418117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:26.559832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:26.561884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:26.590203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:26.597278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:26.638612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1101/2187] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:26 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 533 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:26.709949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:26.711425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:26.731731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc0f941-abfa-4f0d-ba9f-5eaf59334f62 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:26.743733 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:26.744374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:26.744699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:26.745258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:26.745651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4797c4-3ac0-4254-9787-1423c5a91e8a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:26.749428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:26.750515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:26.751258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:26.751489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:26.751766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] PATH_INFO: `/v3/projects/f9c6ed7fd15d46ceaf45c31ff5431d64` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:26.752993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9c6ed7fd15d46ceaf45c31ff5431d64)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:26.755119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:26.871109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9c6ed7fd15d46ceaf45c31ff5431d64', 'request_id': 'req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.898610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9c6ed7fd15d46ceaf45c31ff5431d64', 'request_id': 'req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.912149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:26.915061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:26.922440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9c6ed7fd15d46ceaf45c31ff5431d64', 'request_id': 'req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.954193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1102/2188] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:26 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:26.957530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:26.957741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] The token cache is being invalidate because project f9c6ed7fd15d46ceaf45c31ff5431d64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:26.958300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.960833 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8d10760e-6c3f-4308-a25b-7dba4dcaa36c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:26.963810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:26.969633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1087/2189] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:26 2022] DELETE /identity/v3/projects/f9c6ed7fd15d46ceaf45c31ff5431d64 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:26.973588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39559b14-693d-4272-bbd8-9376af44152a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:26.975653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39559b14-693d-4272-bbd8-9376af44152a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:26.975958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39559b14-693d-4272-bbd8-9376af44152a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.001298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e2eced-bfee-4d52-86e6-e19243ee7c45 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:27.028152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-39559b14-693d-4272-bbd8-9376af44152a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:27.049850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-39559b14-693d-4272-bbd8-9376af44152a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:27.117989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:27.133742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:27.134263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:27.134544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:27.135595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.139609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:27.170546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1088/2190] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:27.179422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7d0cf-130e-4636-a365-cea38f166612 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:27.181726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7d0cf-130e-4636-a365-cea38f166612 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:27.182812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7d0cf-130e-4636-a365-cea38f166612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.222277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-88b7d0cf-130e-4636-a365-cea38f166612 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:27.223751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-88b7d0cf-130e-4636-a365-cea38f166612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:27.236815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1089/2191] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:27.241246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b7d0cf-130e-4636-a365-cea38f166612 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:27.252905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b7d0cf-130e-4636-a365-cea38f166612 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:27.257545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:27.259501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:27.264241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:27.266873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:27.267360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:27.268096 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.269829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:27.270424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:27.273362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:27.288698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1103/2192] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:26 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:27.293955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:27.299988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1090/2193] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:42:27.307711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:27.332652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc00e5e0-72c3-49f8-a5c2-20c8ff9f608e service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:27.347287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6856a986-3f54-4b77-bb6f-dd0fc733264e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:27.347906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6856a986-3f54-4b77-bb6f-dd0fc733264e admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:27.348715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6856a986-3f54-4b77-bb6f-dd0fc733264e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:27.348715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6856a986-3f54-4b77-bb6f-dd0fc733264e admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.349136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6856a986-3f54-4b77-bb6f-dd0fc733264e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:27.351946 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6856a986-3f54-4b77-bb6f-dd0fc733264e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:27.427632 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1104/2194] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:27 2022] POST /identity/v3/projects => generated 367 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:27.435418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:27.459827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b92f0-2697-4a81-9c44-c3062a2a219b admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:27.723532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1f474a-ff4f-4002-9646-ba6f170edf18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:27.724364 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f474a-ff4f-4002-9646-ba6f170edf18 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:27.724495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f474a-ff4f-4002-9646-ba6f170edf18 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:27.725346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f474a-ff4f-4002-9646-ba6f170edf18 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.726222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1f474a-ff4f-4002-9646-ba6f170edf18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:27.739008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1f474a-ff4f-4002-9646-ba6f170edf18 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:27.777790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1091/2195] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:27 2022] POST /identity/v3/users => generated 284 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:27.782775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6856a986-3f54-4b77-bb6f-dd0fc733264e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:27.796882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6856a986-3f54-4b77-bb6f-dd0fc733264e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:27.803805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb6c38b-115d-4510-8fc6-dbeada46ae9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:27.804688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb6c38b-115d-4510-8fc6-dbeada46ae9f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:27.805167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb6c38b-115d-4510-8fc6-dbeada46ae9f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:27.805661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb6c38b-115d-4510-8fc6-dbeada46ae9f admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.806564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb6c38b-115d-4510-8fc6-dbeada46ae9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:27.809296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb6c38b-115d-4510-8fc6-dbeada46ae9f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:27.828720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1105/2196] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:27 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:27.833395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1f474a-ff4f-4002-9646-ba6f170edf18 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:27.853592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1f474a-ff4f-4002-9646-ba6f170edf18 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:27.865381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9e3513-4c53-4371-8d1e-8dbefca7dfef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:27.866349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e3513-4c53-4371-8d1e-8dbefca7dfef admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:27.866349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e3513-4c53-4371-8d1e-8dbefca7dfef admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:27.867484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e3513-4c53-4371-8d1e-8dbefca7dfef admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.867913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9e3513-4c53-4371-8d1e-8dbefca7dfef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:27.871070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9e3513-4c53-4371-8d1e-8dbefca7dfef admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:27.887417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1092/2197] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:27 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:27.893526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb6c38b-115d-4510-8fc6-dbeada46ae9f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:27.928404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb6c38b-115d-4510-8fc6-dbeada46ae9f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:27.939744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:27.940478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:27.940857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:27.941202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] PATH_INFO: `/v3/projects/5f112e3a335046ea947e302c1208bf07/users/e21d5b6e9c674459a4b9e425a7b89474/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:27.968070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f112e3a335046ea947e302c1208bf07, user_id=e21d5b6e9c674459a4b9e425a7b89474, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:27.971218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:27.996571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1106/2198] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:27 2022] PUT /identity/v3/projects/5f112e3a335046ea947e302c1208bf07/users/e21d5b6e9c674459a4b9e425a7b89474/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:28.008779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65363eb8-d288-4928-a2a2-1a5d19755ca9 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:28.009067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65363eb8-d288-4928-a2a2-1a5d19755ca9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:28.009340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65363eb8-d288-4928-a2a2-1a5d19755ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:28.041252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-65363eb8-d288-4928-a2a2-1a5d19755ca9 None None] MFA Rules not processed for user `e21d5b6e9c674459a4b9e425a7b89474`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:28.044820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-65363eb8-d288-4928-a2a2-1a5d19755ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:28.132743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1093/2199] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:28 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:28.147907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:28.150655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:28.275527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49aa8d95-8429-407c-aa33-63df281e99bf admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:28.280506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab4893-370d-4a9c-8292-eeda13a13b7e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:28.281676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab4893-370d-4a9c-8292-eeda13a13b7e service glance] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:28.281975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab4893-370d-4a9c-8292-eeda13a13b7e service glance] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:28.282259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab4893-370d-4a9c-8292-eeda13a13b7e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:28.302100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcab4893-370d-4a9c-8292-eeda13a13b7e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:28.303726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcab4893-370d-4a9c-8292-eeda13a13b7e service glance] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:28.319107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1107/2200] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:28 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:31.547112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65363eb8-d288-4928-a2a2-1a5d19755ca9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:31.548775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-65363eb8-d288-4928-a2a2-1a5d19755ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:31.641181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65363eb8-d288-4928-a2a2-1a5d19755ca9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:31.645816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:31.646463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:31.646761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:31.647112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] PATH_INFO: `/v3/users/d6dd5a84af3e4e22b80b5a068cd2cca3` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:31.648464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6dd5a84af3e4e22b80b5a068cd2cca3)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:31.650407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:31.657265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:31.717551 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6dd5a84af3e4e22b80b5a068cd2cca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:31.718245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6dd5a84af3e4e22b80b5a068cd2cca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:31.727033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6dd5a84af3e4e22b80b5a068cd2cca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:31.736305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6dd5a84af3e4e22b80b5a068cd2cca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:31.752102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6dd5a84af3e4e22b80b5a068cd2cca3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:31.765146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1094/2201] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:31 2022] DELETE /identity/v3/users/d6dd5a84af3e4e22b80b5a068cd2cca3 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:31.769758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa96f50-e124-41d3-a7ab-a418fed8ab15 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:31.770292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa96f50-e124-41d3-a7ab-a418fed8ab15 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:31.770445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa96f50-e124-41d3-a7ab-a418fed8ab15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:31.802980 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-cfa96f50-e124-41d3-a7ab-a418fed8ab15 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:31.804220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cfa96f50-e124-41d3-a7ab-a418fed8ab15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:31.821080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:31.883099 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1108/2202] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:31.901001 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:31.913778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:31.998683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc43f0-9665-4a0f-843b-4b215fb499a4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:32.003039 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08057296-1a76-4d36-8736-3957d9c1165a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:32.003879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08057296-1a76-4d36-8736-3957d9c1165a service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:32.004318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08057296-1a76-4d36-8736-3957d9c1165a service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:32.004769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08057296-1a76-4d36-8736-3957d9c1165a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:32.015479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08057296-1a76-4d36-8736-3957d9c1165a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:32.017317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08057296-1a76-4d36-8736-3957d9c1165a service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:32.029363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1095/2203] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:32.337354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa96f50-e124-41d3-a7ab-a418fed8ab15 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:32.339528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cfa96f50-e124-41d3-a7ab-a418fed8ab15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:32.353168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa96f50-e124-41d3-a7ab-a418fed8ab15 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:32.358626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:32.359578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:32.359933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:32.360300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] PATH_INFO: `/v3/users/8fe5792e5fc849cf853a600426e41079` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:32.361529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fe5792e5fc849cf853a600426e41079)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:32.363586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:32.426106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fe5792e5fc849cf853a600426e41079'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.434403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fe5792e5fc849cf853a600426e41079'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.435182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fe5792e5fc849cf853a600426e41079'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.436108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fe5792e5fc849cf853a600426e41079'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.451851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fe5792e5fc849cf853a600426e41079'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.473355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1109/2204] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:32 2022] DELETE /identity/v3/users/8fe5792e5fc849cf853a600426e41079 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:32.478038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b8c134-5c90-4833-8cad-065685094eb1 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:32.478309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b8c134-5c90-4833-8cad-065685094eb1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:32.478594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b8c134-5c90-4833-8cad-065685094eb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:32.508113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-00b8c134-5c90-4833-8cad-065685094eb1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:32.510154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-00b8c134-5c90-4833-8cad-065685094eb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:32.638687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1096/2205] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:32.656693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:32.658290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:32.735378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b8c134-5c90-4833-8cad-065685094eb1 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:32.736836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-00b8c134-5c90-4833-8cad-065685094eb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:32.738779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616d46b9-942c-4f5b-a5f2-5f33b9e8fe0d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:32.744610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17941a3c-ad85-48b4-84ab-ac00d49b6f14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:32.745615 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941a3c-ad85-48b4-84ab-ac00d49b6f14 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:32.746159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941a3c-ad85-48b4-84ab-ac00d49b6f14 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:32.746646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941a3c-ad85-48b4-84ab-ac00d49b6f14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:32.762008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17941a3c-ad85-48b4-84ab-ac00d49b6f14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:32.763500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17941a3c-ad85-48b4-84ab-ac00d49b6f14 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:32.769542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b8c134-5c90-4833-8cad-065685094eb1 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:32.774243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1110/2206] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:32.776054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:32.777269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:32.777635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:32.777920 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] PATH_INFO: `/v3/projects/bdb8eca30d6849ccb89066cea399b74a` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:32.779076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdb8eca30d6849ccb89066cea399b74a)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:32.781211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:32.820861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:32.861923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdb8eca30d6849ccb89066cea399b74a', 'request_id': 'req-8977b0ad-503f-4a6e-ae95-f462803963d5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.870345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdb8eca30d6849ccb89066cea399b74a', 'request_id': 'req-8977b0ad-503f-4a6e-ae95-f462803963d5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.880598 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdb8eca30d6849ccb89066cea399b74a', 'request_id': 'req-8977b0ad-503f-4a6e-ae95-f462803963d5'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.894690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] The token cache is being invalidate because project bdb8eca30d6849ccb89066cea399b74a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:32.895308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.896194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:32.898033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1097/2207] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:32 2022] DELETE /identity/v3/projects/bdb8eca30d6849ccb89066cea399b74a => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:32.902829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17941a3c-ad85-48b4-84ab-ac00d49b6f14 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:32.915009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17941a3c-ad85-48b4-84ab-ac00d49b6f14 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:32.965810 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:33.002909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.004014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.004852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.005244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] PATH_INFO: `/v3/users/fbb9d3507a3e43a1a3cd2f2dcb6dea70` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.006547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbb9d3507a3e43a1a3cd2f2dcb6dea70)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.008648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.061105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbb9d3507a3e43a1a3cd2f2dcb6dea70'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.068325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbb9d3507a3e43a1a3cd2f2dcb6dea70'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.069107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbb9d3507a3e43a1a3cd2f2dcb6dea70'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.069805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbb9d3507a3e43a1a3cd2f2dcb6dea70'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.078378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbb9d3507a3e43a1a3cd2f2dcb6dea70'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.097989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1111/2208] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:32 2022] DELETE /identity/v3/users/fbb9d3507a3e43a1a3cd2f2dcb6dea70 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:33.117508 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:33.119146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:33.133518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8977b0ad-503f-4a6e-ae95-f462803963d5 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:33.212152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.212791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.213074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.213353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] PATH_INFO: `/v3/projects/86c655d027dd4b59a6752e5abe9e2bf0` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.214576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=86c655d027dd4b59a6752e5abe9e2bf0)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.216562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.268110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86c655d027dd4b59a6752e5abe9e2bf0', 'request_id': 'req-08aeb6e7-8b40-416a-9252-333396754f1c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.275361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86c655d027dd4b59a6752e5abe9e2bf0', 'request_id': 'req-08aeb6e7-8b40-416a-9252-333396754f1c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.284714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86c655d027dd4b59a6752e5abe9e2bf0', 'request_id': 'req-08aeb6e7-8b40-416a-9252-333396754f1c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.301195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] The token cache is being invalidate because project 86c655d027dd4b59a6752e5abe9e2bf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:33.301654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.303043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.305736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1098/2209] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:33 2022] DELETE /identity/v3/projects/86c655d027dd4b59a6752e5abe9e2bf0 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:33.309690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:33.311432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:33.327149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b3467-9166-4ef3-8fd6-1bb8f487b366 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:33.412156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.412498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.412808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.413080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] PATH_INFO: `/v3/users/05a8e834732f4c35aa567adddaa11447` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.414221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05a8e834732f4c35aa567adddaa11447)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.416218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.439410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:33.441157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:33.463397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aeb6e7-8b40-416a-9252-333396754f1c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:33.469699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.470345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.470617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.470928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] PATH_INFO: `/v3/projects/e6a8dee9fb0c44efaa3f5932b1d7a00f` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.472086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6a8dee9fb0c44efaa3f5932b1d7a00f)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.474013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.487470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05a8e834732f4c35aa567adddaa11447'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.496371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05a8e834732f4c35aa567adddaa11447'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.497097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05a8e834732f4c35aa567adddaa11447'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.498388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05a8e834732f4c35aa567adddaa11447'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.500281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:33.515504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cd6943a3-17dd-4bc5-9525-b9ebb37cddc7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05a8e834732f4c35aa567adddaa11447'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.544763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:33.545014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1112/2210] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:33 2022] DELETE /identity/v3/users/05a8e834732f4c35aa567adddaa11447 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:33.558242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6a8dee9fb0c44efaa3f5932b1d7a00f', 'request_id': 'req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.564689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6a8dee9fb0c44efaa3f5932b1d7a00f', 'request_id': 'req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.573575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6a8dee9fb0c44efaa3f5932b1d7a00f', 'request_id': 'req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.585434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] The token cache is being invalidate because project e6a8dee9fb0c44efaa3f5932b1d7a00f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:33.585703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.586399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4bb5cdc2-39ff-482a-91c1-6260a89fc64c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.588337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1099/2211] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:33 2022] DELETE /identity/v3/projects/e6a8dee9fb0c44efaa3f5932b1d7a00f => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:33.593589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1497b52-b779-47fc-90a9-7abc46cf658c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.593928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1497b52-b779-47fc-90a9-7abc46cf658c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.594369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1497b52-b779-47fc-90a9-7abc46cf658c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.617102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c1497b52-b779-47fc-90a9-7abc46cf658c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:33.619344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c1497b52-b779-47fc-90a9-7abc46cf658c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:33.707403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1113/2212] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:33.712383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38280a33-33e6-4d9f-8b99-4f4043296997 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.713377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38280a33-33e6-4d9f-8b99-4f4043296997 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.713659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38280a33-33e6-4d9f-8b99-4f4043296997 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.738846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-38280a33-33e6-4d9f-8b99-4f4043296997 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:33.740159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-38280a33-33e6-4d9f-8b99-4f4043296997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:33.753657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1100/2213] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:33.754618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1497b52-b779-47fc-90a9-7abc46cf658c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:33.765115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1497b52-b779-47fc-90a9-7abc46cf658c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:33.769913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.770562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.770878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.771152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.771813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:33.772121 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.773726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.786363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1114/2214] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:42:33.790802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38280a33-33e6-4d9f-8b99-4f4043296997 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:33.802567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38280a33-33e6-4d9f-8b99-4f4043296997 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:33.808613 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.809305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.809654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.809998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.810698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.812830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.845277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1101/2215] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:33 2022] POST /identity/v3/projects => generated 361 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:33.849628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:33.855225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:33.857790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:33.864146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0570acf4-2407-4b68-b89d-f30a61f8e185 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:33.928349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6692815-5aa5-42c1-ade6-9abec08c9fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.928876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6692815-5aa5-42c1-ade6-9abec08c9fc4 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.929159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6692815-5aa5-42c1-ade6-9abec08c9fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.929432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6692815-5aa5-42c1-ade6-9abec08c9fc4 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.929674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd2fc3f-57b9-4c7b-9874-827dbb8b6f73 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:33.930085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6692815-5aa5-42c1-ade6-9abec08c9fc4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.932070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6692815-5aa5-42c1-ade6-9abec08c9fc4 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.935049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.935696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.935978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.936252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] PATH_INFO: `/v3/projects/ffe4fcc4bba142ef9e608278fdae7443` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.938040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffe4fcc4bba142ef9e608278fdae7443)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.940017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.957206 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1115/2216] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:33 2022] POST /identity/v3/users => generated 281 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:33.961132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6692815-5aa5-42c1-ade6-9abec08c9fc4 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:33.972391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6692815-5aa5-42c1-ade6-9abec08c9fc4 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:33.977268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4b1bf8-f205-49e7-8da8-d1b4b5bc14a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:33.977906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4b1bf8-f205-49e7-8da8-d1b4b5bc14a0 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:33.978192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4b1bf8-f205-49e7-8da8-d1b4b5bc14a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:33.978466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4b1bf8-f205-49e7-8da8-d1b4b5bc14a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:33.980663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4b1bf8-f205-49e7-8da8-d1b4b5bc14a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:33.982331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4b1bf8-f205-49e7-8da8-d1b4b5bc14a0 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:33.992325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffe4fcc4bba142ef9e608278fdae7443', 'request_id': 'req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:33.999876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1116/2217] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:33 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:34.004094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4b1bf8-f205-49e7-8da8-d1b4b5bc14a0 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:34.005415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffe4fcc4bba142ef9e608278fdae7443', 'request_id': 'req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.021341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4b1bf8-f205-49e7-8da8-d1b4b5bc14a0 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:34.023226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffe4fcc4bba142ef9e608278fdae7443', 'request_id': 'req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.035934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] The token cache is being invalidate because project ffe4fcc4bba142ef9e608278fdae7443 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:34.036464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.036880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.038409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1102/2218] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:33 2022] DELETE /identity/v3/projects/ffe4fcc4bba142ef9e608278fdae7443 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:34.043005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:34.044449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:34.070928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:34.079238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:34.095509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:34.096124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.096406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.096687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.097339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:34.098990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:34.109780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1117/2219] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:34 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:34.114176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:34.116087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:34.118795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f56d566-d3e4-4972-b7f7-a51a195f6e4a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:34.126766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:34.126766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.126766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.126766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] PATH_INFO: `/v3/users/b64422f1c0864c6ebfe89a839b46e948` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.127402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b64422f1c0864c6ebfe89a839b46e948)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:34.129371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:34.130304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a53525-1c03-41e8-bd80-c58300534161 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:34.135337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c65c34-27cd-49fd-8c96-d3cc0b409a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:34.136226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c65c34-27cd-49fd-8c96-d3cc0b409a53 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.136677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c65c34-27cd-49fd-8c96-d3cc0b409a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.136970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c65c34-27cd-49fd-8c96-d3cc0b409a53 admin admin] PATH_INFO: `/v3/projects/d10d17e1a1554981b8d24af5eea8b174/users/a97d7271891c4f5db76ec16744950ecf/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.153157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c65c34-27cd-49fd-8c96-d3cc0b409a53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d10d17e1a1554981b8d24af5eea8b174, user_id=a97d7271891c4f5db76ec16744950ecf, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:34.156021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c65c34-27cd-49fd-8c96-d3cc0b409a53 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:34.170554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1118/2220] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:34 2022] PUT /identity/v3/projects/d10d17e1a1554981b8d24af5eea8b174/users/a97d7271891c4f5db76ec16744950ecf/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:34.178890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b64422f1c0864c6ebfe89a839b46e948'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.179499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b64422f1c0864c6ebfe89a839b46e948'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.181087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8b5f30-24f9-4823-a741-e96568ac8a93 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.181484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8b5f30-24f9-4823-a741-e96568ac8a93 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.181771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8b5f30-24f9-4823-a741-e96568ac8a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.187741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b64422f1c0864c6ebfe89a839b46e948'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.196041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b64422f1c0864c6ebfe89a839b46e948'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.209308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b64422f1c0864c6ebfe89a839b46e948'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.210163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1a8b5f30-24f9-4823-a741-e96568ac8a93 None None] MFA Rules not processed for user `a97d7271891c4f5db76ec16744950ecf`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:34.211425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1a8b5f30-24f9-4823-a741-e96568ac8a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:34.215465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1103/2221] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:34 2022] DELETE /identity/v3/users/b64422f1c0864c6ebfe89a839b46e948 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:34.293759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1119/2222] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:34 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:34.334299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:34.335879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:34.428498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d275c7-1da1-4a71-80fe-2d8dba055e94 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:34.432310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:34.433615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.433972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.434250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.515127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:34.516979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:34.522915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:34.538279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1104/2223] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:34 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:34.554783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8b5f30-24f9-4823-a741-e96568ac8a93 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:34.558342 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1a8b5f30-24f9-4823-a741-e96568ac8a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:34.567645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:34.569627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:34.695826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a40a56-9dcc-48a2-84b2-9b0c0c62755c service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:34.696038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8b5f30-24f9-4823-a741-e96568ac8a93 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:34.700896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:34.701541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.701829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.702127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] PATH_INFO: `/v3/projects/1530ebce582d4378ae92165ca3ba4121` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.703621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:34.703771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] RBAC: Authorizing `identity:delete_project(project_id=1530ebce582d4378ae92165ca3ba4121)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:34.704330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.704617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.704895 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] PATH_INFO: `/v3/users/a97d7271891c4f5db76ec16744950ecf` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.705574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:34.706657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a97d7271891c4f5db76ec16744950ecf)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:34.710001 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:34.773493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1530ebce582d4378ae92165ca3ba4121', 'request_id': 'req-52c1a4fd-db0d-4f87-a3b3-f894d867960b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.780916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1530ebce582d4378ae92165ca3ba4121', 'request_id': 'req-52c1a4fd-db0d-4f87-a3b3-f894d867960b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.799079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1530ebce582d4378ae92165ca3ba4121', 'request_id': 'req-52c1a4fd-db0d-4f87-a3b3-f894d867960b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.803227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:34.806700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a97d7271891c4f5db76ec16744950ecf'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.815983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] The token cache is being invalidate because project 1530ebce582d4378ae92165ca3ba4121 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:34.816630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.817808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a97d7271891c4f5db76ec16744950ecf'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.818523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a97d7271891c4f5db76ec16744950ecf'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.819150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a97d7271891c4f5db76ec16744950ecf'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.831373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-52c1a4fd-db0d-4f87-a3b3-f894d867960b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.837086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1105/2224] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:34 2022] DELETE /identity/v3/projects/1530ebce582d4378ae92165ca3ba4121 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:34.838623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ce2221-4022-40cd-9216-2b1f710d273f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.839103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ce2221-4022-40cd-9216-2b1f710d273f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.839459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ce2221-4022-40cd-9216-2b1f710d273f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.863553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2a61a25a-f64b-4983-91ea-87b2665c11f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a97d7271891c4f5db76ec16744950ecf'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:34.865973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d6ce2221-4022-40cd-9216-2b1f710d273f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:34.867412 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d6ce2221-4022-40cd-9216-2b1f710d273f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:34.893434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1120/2225] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:34 2022] DELETE /identity/v3/users/a97d7271891c4f5db76ec16744950ecf => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:34.897318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40254e1f-eb34-44ce-992c-f510724277c1 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:34.897318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40254e1f-eb34-44ce-992c-f510724277c1 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:34.897318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40254e1f-eb34-44ce-992c-f510724277c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:34.921989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-40254e1f-eb34-44ce-992c-f510724277c1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:34.923133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-40254e1f-eb34-44ce-992c-f510724277c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.032273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1121/2226] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:35.043207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1106/2227] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:35.045034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40254e1f-eb34-44ce-992c-f510724277c1 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.046555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d1fb44-1c27-4f8e-b769-06be751294b3 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.051030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-40254e1f-eb34-44ce-992c-f510724277c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.051709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d1fb44-1c27-4f8e-b769-06be751294b3 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.052665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d1fb44-1c27-4f8e-b769-06be751294b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.090612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-92d1fb44-1c27-4f8e-b769-06be751294b3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:35.092616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-92d1fb44-1c27-4f8e-b769-06be751294b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.114887 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1107/2228] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:35.116417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d1fb44-1c27-4f8e-b769-06be751294b3 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.135120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d1fb44-1c27-4f8e-b769-06be751294b3 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.143488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.144338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40254e1f-eb34-44ce-992c-f510724277c1 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.145829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.146133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.146409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.147116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:35.147426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.149151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a653a375-8da8-43a8-b118-cbd11175570c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.150005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a653a375-8da8-43a8-b118-cbd11175570c service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.150378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.150774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a653a375-8da8-43a8-b118-cbd11175570c service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.151945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a653a375-8da8-43a8-b118-cbd11175570c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.164431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a653a375-8da8-43a8-b118-cbd11175570c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.167805 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1108/2229] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:42:35.169535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a653a375-8da8-43a8-b118-cbd11175570c service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.172436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.189236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1122/2230] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:35.190457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef3638a-90e5-42a7-bbcf-06ceeb084f58 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.197630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2dd94d-127b-4c14-9832-74a03df817c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.198630 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2dd94d-127b-4c14-9832-74a03df817c4 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.199162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2dd94d-127b-4c14-9832-74a03df817c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.199346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2dd94d-127b-4c14-9832-74a03df817c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.200194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2dd94d-127b-4c14-9832-74a03df817c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.202706 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2dd94d-127b-4c14-9832-74a03df817c4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.217456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:35.240664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1109/2231] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:35 2022] POST /identity/v3/projects => generated 371 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:35.245039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a653a375-8da8-43a8-b118-cbd11175570c service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.257034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a653a375-8da8-43a8-b118-cbd11175570c service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.348240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336a492-500e-4c26-a3da-596ae82c0120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.348977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336a492-500e-4c26-a3da-596ae82c0120 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.349307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336a492-500e-4c26-a3da-596ae82c0120 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.349671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336a492-500e-4c26-a3da-596ae82c0120 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.351184 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336a492-500e-4c26-a3da-596ae82c0120 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.353302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336a492-500e-4c26-a3da-596ae82c0120 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.364511 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:35.379952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1123/2232] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:35 2022] POST /identity/v3/users => generated 286 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:35.384411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2dd94d-127b-4c14-9832-74a03df817c4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.396331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2dd94d-127b-4c14-9832-74a03df817c4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.403495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71726ecc-f6cc-425e-9258-16a2bb47caca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.404381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71726ecc-f6cc-425e-9258-16a2bb47caca admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.404559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71726ecc-f6cc-425e-9258-16a2bb47caca admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.404914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71726ecc-f6cc-425e-9258-16a2bb47caca admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.405790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71726ecc-f6cc-425e-9258-16a2bb47caca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.408253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71726ecc-f6cc-425e-9258-16a2bb47caca admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.423181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1110/2233] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:35 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:35.424727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336a492-500e-4c26-a3da-596ae82c0120 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.449324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336a492-500e-4c26-a3da-596ae82c0120 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.455165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.455886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.456275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.456675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.457442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.459198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.476048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1124/2234] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:35 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:35.476179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71726ecc-f6cc-425e-9258-16a2bb47caca admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.486939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71726ecc-f6cc-425e-9258-16a2bb47caca admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.493458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf3271b-9ea7-484f-ac7f-75f69ba977ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.494500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf3271b-9ea7-484f-ac7f-75f69ba977ee admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.494621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf3271b-9ea7-484f-ac7f-75f69ba977ee admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.495771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf3271b-9ea7-484f-ac7f-75f69ba977ee admin admin] PATH_INFO: `/v3/projects/ee8671b8d3024f6aac49368289f26eb7/users/a4674e80ea68459499781e922caa33a9/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.515267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf3271b-9ea7-484f-ac7f-75f69ba977ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee8671b8d3024f6aac49368289f26eb7, user_id=a4674e80ea68459499781e922caa33a9, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.518175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf3271b-9ea7-484f-ac7f-75f69ba977ee admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.532085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1111/2235] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:35 2022] PUT /identity/v3/projects/ee8671b8d3024f6aac49368289f26eb7/users/a4674e80ea68459499781e922caa33a9/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:35.543508 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.544264 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a2cfd-7a3a-457d-86f9-f98a1be4f49f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.544574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a2cfd-7a3a-457d-86f9-f98a1be4f49f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.544860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a2cfd-7a3a-457d-86f9-f98a1be4f49f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.546216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.561010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bbd5f4-9589-4e60-834a-12fe435ba58e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.590180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1f3a2cfd-7a3a-457d-86f9-f98a1be4f49f None None] MFA Rules not processed for user `a4674e80ea68459499781e922caa33a9`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:35.591751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1f3a2cfd-7a3a-457d-86f9-f98a1be4f49f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.667873 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.668601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.668877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.669149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] PATH_INFO: `/v3/projects/d10d17e1a1554981b8d24af5eea8b174` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.671449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=d10d17e1a1554981b8d24af5eea8b174)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.674942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.682797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1112/2236] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:35 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:35.702265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3a2cfd-7a3a-457d-86f9-f98a1be4f49f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.704303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1f3a2cfd-7a3a-457d-86f9-f98a1be4f49f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.752024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd10d17e1a1554981b8d24af5eea8b174', 'request_id': 'req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:35.756879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd10d17e1a1554981b8d24af5eea8b174', 'request_id': 'req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:35.766882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd10d17e1a1554981b8d24af5eea8b174', 'request_id': 'req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:35.785997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] The token cache is being invalidate because project d10d17e1a1554981b8d24af5eea8b174 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:35.786608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:35.789918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8a0054e0-35ef-44e6-9004-14ef3d1e57aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:35.793191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1125/2237] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:35 2022] DELETE /identity/v3/projects/d10d17e1a1554981b8d24af5eea8b174 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:35.797092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b9f152-23cb-492e-9343-0af0701383e5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.797308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b9f152-23cb-492e-9343-0af0701383e5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.797571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b9f152-23cb-492e-9343-0af0701383e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.806045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3a2cfd-7a3a-457d-86f9-f98a1be4f49f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.829299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-30b9f152-23cb-492e-9343-0af0701383e5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:35.830658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-30b9f152-23cb-492e-9343-0af0701383e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.885017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.886176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.886457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.886767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.888550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.903052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:35.914956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1126/2238] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:35.920523 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c828f-5b65-4194-9271-be884a061e0c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.920994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c828f-5b65-4194-9271-be884a061e0c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.921343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c828f-5b65-4194-9271-be884a061e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.943867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e55c828f-5b65-4194-9271-be884a061e0c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:35.945225 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e55c828f-5b65-4194-9271-be884a061e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:35.958206 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1127/2239] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:35.959661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c828f-5b65-4194-9271-be884a061e0c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:35.970457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c828f-5b65-4194-9271-be884a061e0c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:35.971715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.973193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.975416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:35.976039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:35.976321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:35.976591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:35.977364 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:35.977671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:35.979354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:35.980937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:35.982515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1113/2240] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:35 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:35.993094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1128/2241] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:42:35.998240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:36.009866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e08799-1409-4552-bbfa-4b4d6cbdabf0 service glance] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:36.015187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd52840-e435-4a2d-ae01-f121736707cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:36.015788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd52840-e435-4a2d-ae01-f121736707cb admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:36.016064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd52840-e435-4a2d-ae01-f121736707cb admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:36.016333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd52840-e435-4a2d-ae01-f121736707cb admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:36.016963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd52840-e435-4a2d-ae01-f121736707cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:36.018870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd52840-e435-4a2d-ae01-f121736707cb admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:36.049289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1114/2242] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:35 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:36.054262 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:36.065904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92510185-d361-4763-a0ce-ab60056d152e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:36.138041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83adf4b-db4f-4fec-b412-91acaa2a7334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:36.138660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83adf4b-db4f-4fec-b412-91acaa2a7334 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:36.138972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83adf4b-db4f-4fec-b412-91acaa2a7334 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:36.139394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83adf4b-db4f-4fec-b412-91acaa2a7334 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:36.140086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83adf4b-db4f-4fec-b412-91acaa2a7334 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:36.142033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83adf4b-db4f-4fec-b412-91acaa2a7334 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:36.164067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1129/2243] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:36 2022] POST /identity/v3/users => generated 288 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:36.168221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd52840-e435-4a2d-ae01-f121736707cb admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:36.179586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd52840-e435-4a2d-ae01-f121736707cb admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:36.186414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8b5898-c41a-4a67-9b2c-96015cf31727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:36.187323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8b5898-c41a-4a67-9b2c-96015cf31727 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:36.187677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8b5898-c41a-4a67-9b2c-96015cf31727 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:36.188009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8b5898-c41a-4a67-9b2c-96015cf31727 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:36.188743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8b5898-c41a-4a67-9b2c-96015cf31727 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:36.190506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8b5898-c41a-4a67-9b2c-96015cf31727 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:36.203740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1115/2244] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:36.207331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83adf4b-db4f-4fec-b412-91acaa2a7334 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:36.217781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83adf4b-db4f-4fec-b412-91acaa2a7334 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:36.224310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe702af1-fed5-43d5-9f80-bc9b14356fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:36.225008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe702af1-fed5-43d5-9f80-bc9b14356fef admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:36.225293 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe702af1-fed5-43d5-9f80-bc9b14356fef admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:36.225596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe702af1-fed5-43d5-9f80-bc9b14356fef admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:36.226231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe702af1-fed5-43d5-9f80-bc9b14356fef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:36.227976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe702af1-fed5-43d5-9f80-bc9b14356fef admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:36.253201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1130/2245] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:36 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:36.254334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8b5898-c41a-4a67-9b2c-96015cf31727 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:36.265133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8b5898-c41a-4a67-9b2c-96015cf31727 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:36.270240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:36.271155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:36.271897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:36.271897 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] PATH_INFO: `/v3/projects/3dc7588b0cd64daf8b4af6cca3f73018/users/df3a85632eb94ef3b22981cb2da51b0d/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:36.288195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3dc7588b0cd64daf8b4af6cca3f73018, user_id=df3a85632eb94ef3b22981cb2da51b0d, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:36.291241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:36.306119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1116/2246] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:36 2022] PUT /identity/v3/projects/3dc7588b0cd64daf8b4af6cca3f73018/users/df3a85632eb94ef3b22981cb2da51b0d/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:42:36.317423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4388c9-90bd-4322-9b7d-a2d616b80662 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:36.317711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4388c9-90bd-4322-9b7d-a2d616b80662 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:36.317995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4388c9-90bd-4322-9b7d-a2d616b80662 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:36.352617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-9c4388c9-90bd-4322-9b7d-a2d616b80662 None None] MFA Rules not processed for user `df3a85632eb94ef3b22981cb2da51b0d`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:36.353891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9c4388c9-90bd-4322-9b7d-a2d616b80662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:36.456560 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1131/2247] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:36 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:36.482607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:36.484552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:36.595873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57a22e4-f113-4524-aadf-7fc74448961d admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:36.600481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47361cc-49c2-4e8d-9923-88190ff821d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:36.601088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47361cc-49c2-4e8d-9923-88190ff821d8 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:36.601369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47361cc-49c2-4e8d-9923-88190ff821d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:36.601636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47361cc-49c2-4e8d-9923-88190ff821d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:36.610902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47361cc-49c2-4e8d-9923-88190ff821d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:36.612362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47361cc-49c2-4e8d-9923-88190ff821d8 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:36.621678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1117/2248] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:36 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:39.821598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4388c9-90bd-4322-9b7d-a2d616b80662 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:39.892485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4388c9-90bd-4322-9b7d-a2d616b80662 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:39.897100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:39.897742 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:39.898023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:39.898300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] PATH_INFO: `/v3/users/df3a85632eb94ef3b22981cb2da51b0d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:39.899456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df3a85632eb94ef3b22981cb2da51b0d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:39.901938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:39.904344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:39.953384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df3a85632eb94ef3b22981cb2da51b0d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:39.961600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df3a85632eb94ef3b22981cb2da51b0d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:39.962344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df3a85632eb94ef3b22981cb2da51b0d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:39.962996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df3a85632eb94ef3b22981cb2da51b0d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:39.972264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df3a85632eb94ef3b22981cb2da51b0d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:39.990448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1132/2249] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:42:39 2022] DELETE /identity/v3/users/df3a85632eb94ef3b22981cb2da51b0d => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:39.995091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7089731a-801f-43b0-b595-faa103043748 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:39.995635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7089731a-801f-43b0-b595-faa103043748 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:39.995635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7089731a-801f-43b0-b595-faa103043748 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:40.015074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7089731a-801f-43b0-b595-faa103043748 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:40.016280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7089731a-801f-43b0-b595-faa103043748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:40.070066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:40.081006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1118/2250] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:40.091734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:40.093265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:40.156836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb788464-3bab-41b1-94cc-c195f90e0f95 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:40.161661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6806d3-1061-4e9a-b043-698f8525650e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:40.162327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6806d3-1061-4e9a-b043-698f8525650e service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:40.162614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6806d3-1061-4e9a-b043-698f8525650e service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:40.162927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6806d3-1061-4e9a-b043-698f8525650e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:40.172928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6806d3-1061-4e9a-b043-698f8525650e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:40.174504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6806d3-1061-4e9a-b043-698f8525650e service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:40.185197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1133/2251] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:40.465396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7089731a-801f-43b0-b595-faa103043748 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:40.467210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7089731a-801f-43b0-b595-faa103043748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:40.480543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7089731a-801f-43b0-b595-faa103043748 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:40.486316 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:40.487115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:40.487396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:40.487709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] PATH_INFO: `/v3/projects/3dc7588b0cd64daf8b4af6cca3f73018` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:40.493937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3dc7588b0cd64daf8b4af6cca3f73018)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:40.496432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:40.552722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3dc7588b0cd64daf8b4af6cca3f73018', 'request_id': 'req-53187d82-3a96-401e-9c43-8bcb195bd66c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:40.559752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3dc7588b0cd64daf8b4af6cca3f73018', 'request_id': 'req-53187d82-3a96-401e-9c43-8bcb195bd66c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:40.570491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3dc7588b0cd64daf8b4af6cca3f73018', 'request_id': 'req-53187d82-3a96-401e-9c43-8bcb195bd66c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:40.584128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] The token cache is being invalidate because project 3dc7588b0cd64daf8b4af6cca3f73018 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:42:40.584556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:40.585480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-53187d82-3a96-401e-9c43-8bcb195bd66c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:42:40.587495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1119/2252] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:42:40 2022] DELETE /identity/v3/projects/3dc7588b0cd64daf8b4af6cca3f73018 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:40.595669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca1cd9-d522-45b4-b41d-eaf4be64527d None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:40.596082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca1cd9-d522-45b4-b41d-eaf4be64527d None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:40.596369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca1cd9-d522-45b4-b41d-eaf4be64527d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:40.618543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-deca1cd9-d522-45b4-b41d-eaf4be64527d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:40.620416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-deca1cd9-d522-45b4-b41d-eaf4be64527d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:40.726386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1134/2253] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:40.730486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e73460-bbdf-4350-aac2-e1d7e3d1f9d2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:40.730486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e73460-bbdf-4350-aac2-e1d7e3d1f9d2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:40.730486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e73460-bbdf-4350-aac2-e1d7e3d1f9d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:40.765703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a2e73460-bbdf-4350-aac2-e1d7e3d1f9d2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:40.766954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a2e73460-bbdf-4350-aac2-e1d7e3d1f9d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:40.775749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1120/2254] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:40.781146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deca1cd9-d522-45b4-b41d-eaf4be64527d None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:40.796882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deca1cd9-d522-45b4-b41d-eaf4be64527d None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:40.802665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:40.803416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:40.803763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:40.804102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:40.805237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:42:40.805688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:40.808748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:40.831952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1135/2255] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:42:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:42:40.836483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e73460-bbdf-4350-aac2-e1d7e3d1f9d2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:40.846317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e73460-bbdf-4350-aac2-e1d7e3d1f9d2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:40.851077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad881851-d4c4-45c9-864b-ab038845a0c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:40.851682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad881851-d4c4-45c9-864b-ab038845a0c6 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:40.851963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad881851-d4c4-45c9-864b-ab038845a0c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:40.852235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad881851-d4c4-45c9-864b-ab038845a0c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:40.852969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad881851-d4c4-45c9-864b-ab038845a0c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:40.856730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad881851-d4c4-45c9-864b-ab038845a0c6 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:40.890307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1121/2256] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:42:40 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:40.894392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:40.904850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f754b86b-99af-47c1-ae16-10c5d2962b6c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:40.913969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:40.986124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba76e7-b70f-4b5c-b62e-08ea56526387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:40.987245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba76e7-b70f-4b5c-b62e-08ea56526387 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:40.987750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba76e7-b70f-4b5c-b62e-08ea56526387 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:40.988022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba76e7-b70f-4b5c-b62e-08ea56526387 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:40.988832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba76e7-b70f-4b5c-b62e-08ea56526387 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:40.990935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba76e7-b70f-4b5c-b62e-08ea56526387 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:41.023391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1136/2257] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:42:40 2022] POST /identity/v3/users => generated 292 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:42:41.028160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad881851-d4c4-45c9-864b-ab038845a0c6 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:41.038902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad881851-d4c4-45c9-864b-ab038845a0c6 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:41.044184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ed5ab8-1f9a-479f-989e-8c749f181e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:41.044789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed5ab8-1f9a-479f-989e-8c749f181e80 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:41.045071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed5ab8-1f9a-479f-989e-8c749f181e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:41.045376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed5ab8-1f9a-479f-989e-8c749f181e80 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:41.046027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ed5ab8-1f9a-479f-989e-8c749f181e80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:41.047705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ed5ab8-1f9a-479f-989e-8c749f181e80 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:41.062674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1122/2258] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:42:41.066860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba76e7-b70f-4b5c-b62e-08ea56526387 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:41.078428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba76e7-b70f-4b5c-b62e-08ea56526387 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:41.083383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957a7a30-8be8-46e5-ab8a-e94c7f822ecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:41.084096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957a7a30-8be8-46e5-ab8a-e94c7f822ecd admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:41.084527 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957a7a30-8be8-46e5-ab8a-e94c7f822ecd admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:41.084799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957a7a30-8be8-46e5-ab8a-e94c7f822ecd admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:41.085447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957a7a30-8be8-46e5-ab8a-e94c7f822ecd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:41.087174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957a7a30-8be8-46e5-ab8a-e94c7f822ecd admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:41.104467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1137/2259] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:42:41 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:42:41.107794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ed5ab8-1f9a-479f-989e-8c749f181e80 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:41.118266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ed5ab8-1f9a-479f-989e-8c749f181e80 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:41.123553 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:41.124528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:41.124815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:41.125366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] PATH_INFO: `/v3/projects/211213d0c20641adb48a3d39703f7532/users/2d0e45a9ee9644f8b9eac57d9a14d14e/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:41.144248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=211213d0c20641adb48a3d39703f7532, user_id=2d0e45a9ee9644f8b9eac57d9a14d14e, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:41.148112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:41.167943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1123/2260] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:42:41 2022] PUT /identity/v3/projects/211213d0c20641adb48a3d39703f7532/users/2d0e45a9ee9644f8b9eac57d9a14d14e/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:42:41.172649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2966f5-69cf-4b10-8fd8-0e7a2fdec4e6 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:41.173048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2966f5-69cf-4b10-8fd8-0e7a2fdec4e6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:41.173320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2966f5-69cf-4b10-8fd8-0e7a2fdec4e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:41.197422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-ac2966f5-69cf-4b10-8fd8-0e7a2fdec4e6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:41.199222 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ac2966f5-69cf-4b10-8fd8-0e7a2fdec4e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:41.290090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1138/2261] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:41.300780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:41.302632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:41.385618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c2a3e7-5521-4a5b-a7c2-dff15d72fe59 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:41.391056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:41.391757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:41.392099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:41.393175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:41.404850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:41.406567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:41.430543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1124/2262] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:42:42.269333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769cad4e-3536-4aa2-af34-fdcb43acfebe None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:42.269814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769cad4e-3536-4aa2-af34-fdcb43acfebe None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:42.270292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769cad4e-3536-4aa2-af34-fdcb43acfebe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:42.305624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-769cad4e-3536-4aa2-af34-fdcb43acfebe None None] MFA Rules not processed for user `2d0e45a9ee9644f8b9eac57d9a14d14e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:42.307212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-769cad4e-3536-4aa2-af34-fdcb43acfebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:42.417165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1139/2263] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:42.427051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:42:42.428891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:42.468080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:42:42.523903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebef372-36fe-427e-9286-de02cb98d8e3 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:42:42.528087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fcec9a-3ee2-4277-ba3e-dad5410a7973 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:42:42.528690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fcec9a-3ee2-4277-ba3e-dad5410a7973 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:42.528973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fcec9a-3ee2-4277-ba3e-dad5410a7973 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:42.529245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fcec9a-3ee2-4277-ba3e-dad5410a7973 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:42.542984 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fcec9a-3ee2-4277-ba3e-dad5410a7973 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:42:42.544224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fcec9a-3ee2-4277-ba3e-dad5410a7973 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:42:42.553677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1125/2264] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:42:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:42:44.038721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5556-a509-4128-9e7f-eaf754c5a1b1 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:44.039203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5556-a509-4128-9e7f-eaf754c5a1b1 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:44.039292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5556-a509-4128-9e7f-eaf754c5a1b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:44.060654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2c9d5556-a509-4128-9e7f-eaf754c5a1b1 None None] MFA Rules not processed for user `2d0e45a9ee9644f8b9eac57d9a14d14e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:44.061877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2c9d5556-a509-4128-9e7f-eaf754c5a1b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:44.069078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1140/2265] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:44 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:42:52.232458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8d768-51d1-433c-9400-5af8b9fed195 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:42:52.233820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8d768-51d1-433c-9400-5af8b9fed195 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:42:52.234400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8d768-51d1-433c-9400-5af8b9fed195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:42:52.262917 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-0fe8d768-51d1-433c-9400-5af8b9fed195 None None] MFA Rules not processed for user `cc7aa14777ab43a083b3a120c4349d67`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:42:52.265137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0fe8d768-51d1-433c-9400-5af8b9fed195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:42:52.350781 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1126/2266] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:42:52 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:42:54.054242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:02.342813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=41 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:09.244605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9b90d-3be8-4772-947e-cc47d0b22563 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:09.245648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9b90d-3be8-4772-947e-cc47d0b22563 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:09.246115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9b90d-3be8-4772-947e-cc47d0b22563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:09.270752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-62b9b90d-3be8-4772-947e-cc47d0b22563 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:09.272107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-62b9b90d-3be8-4772-947e-cc47d0b22563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:09.424195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1141/2267] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:43:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:09.514124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe8d768-51d1-433c-9400-5af8b9fed195 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:09.518657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0fe8d768-51d1-433c-9400-5af8b9fed195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:09.666278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe8d768-51d1-433c-9400-5af8b9fed195 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:09.690154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36b9bd5-8f93-4602-914e-a7ce3d809eb3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:09.691026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36b9bd5-8f93-4602-914e-a7ce3d809eb3 service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:09.691311 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36b9bd5-8f93-4602-914e-a7ce3d809eb3 service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:09.691576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36b9bd5-8f93-4602-914e-a7ce3d809eb3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:09.700933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36b9bd5-8f93-4602-914e-a7ce3d809eb3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:09.702907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36b9bd5-8f93-4602-914e-a7ce3d809eb3 service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:09.712978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1127/2268] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:43:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:43:14.084262 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b9b90d-3be8-4772-947e-cc47d0b22563 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:14.086758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-62b9b90d-3be8-4772-947e-cc47d0b22563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:14.139750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b9b90d-3be8-4772-947e-cc47d0b22563 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:14.161022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:14.162774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:14.163486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:14.164175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] PATH_INFO: `/v3/users/e21d5b6e9c674459a4b9e425a7b89474` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:14.167592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e21d5b6e9c674459a4b9e425a7b89474)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:14.189214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:14.385494 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e21d5b6e9c674459a4b9e425a7b89474'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:14.401376 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e21d5b6e9c674459a4b9e425a7b89474'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:14.402681 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e21d5b6e9c674459a4b9e425a7b89474'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:14.403840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e21d5b6e9c674459a4b9e425a7b89474'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:14.461170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e21d5b6e9c674459a4b9e425a7b89474'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:14.572746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1142/2269] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:43:14 2022] DELETE /identity/v3/users/e21d5b6e9c674459a4b9e425a7b89474 => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:43:14.580532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d53f3-0940-480c-8035-085ba5e5126f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:14.581489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d53f3-0940-480c-8035-085ba5e5126f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:14.581489 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d53f3-0940-480c-8035-085ba5e5126f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:14.649732 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-012d53f3-0940-480c-8035-085ba5e5126f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:14.655272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-012d53f3-0940-480c-8035-085ba5e5126f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:14.907088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1128/2270] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:14.917586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:14.919208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:15.042698 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c85f14c-cda2-4993-885e-3735fba6f054 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:15.047545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4efae0-87d9-41b9-bc55-3b01fb0781be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:15.048215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4efae0-87d9-41b9-bc55-3b01fb0781be service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:15.048894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4efae0-87d9-41b9-bc55-3b01fb0781be service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:15.049069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4efae0-87d9-41b9-bc55-3b01fb0781be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:15.064463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4efae0-87d9-41b9-bc55-3b01fb0781be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:15.066673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4efae0-87d9-41b9-bc55-3b01fb0781be service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:15.085219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1143/2271] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:43:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:43:16.005702 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012d53f3-0940-480c-8035-085ba5e5126f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:16.008555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-012d53f3-0940-480c-8035-085ba5e5126f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:16.066902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012d53f3-0940-480c-8035-085ba5e5126f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:16.086010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:16.088978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:16.090785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:16.092794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] PATH_INFO: `/v3/projects/5f112e3a335046ea947e302c1208bf07` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:16.098222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f112e3a335046ea947e302c1208bf07)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:16.103729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:16.247394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f112e3a335046ea947e302c1208bf07', 'request_id': 'req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:16.265511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f112e3a335046ea947e302c1208bf07', 'request_id': 'req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:16.285202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f112e3a335046ea947e302c1208bf07', 'request_id': 'req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:16.301401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] The token cache is being invalidate because project 5f112e3a335046ea947e302c1208bf07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:43:16.302052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:16.303218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cb54aa5e-27a9-4ae7-bd6e-62b4c9aede6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:16.305963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1129/2272] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:43:16 2022] DELETE /identity/v3/projects/5f112e3a335046ea947e302c1208bf07 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:43:16.314265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9d2018-67d4-4ee1-88c4-e87e05ca7f6c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:16.315827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9d2018-67d4-4ee1-88c4-e87e05ca7f6c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:16.319682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9d2018-67d4-4ee1-88c4-e87e05ca7f6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:16.368136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-0a9d2018-67d4-4ee1-88c4-e87e05ca7f6c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:16.371043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0a9d2018-67d4-4ee1-88c4-e87e05ca7f6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:16.400022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:16.522381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1144/2273] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:16.523946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88130ea-5638-45a0-b210-2aac33e5ad18 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:16.524197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88130ea-5638-45a0-b210-2aac33e5ad18 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:16.524365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88130ea-5638-45a0-b210-2aac33e5ad18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:16.573369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e88130ea-5638-45a0-b210-2aac33e5ad18 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:16.574990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e88130ea-5638-45a0-b210-2aac33e5ad18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:16.605722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1130/2274] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:43:16.612460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9d2018-67d4-4ee1-88c4-e87e05ca7f6c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:16.630240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9d2018-67d4-4ee1-88c4-e87e05ca7f6c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:16.636461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:16.637291 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:16.637714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:16.638078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:16.639088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:43:16.639629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:16.642339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:16.661472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1145/2275] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:43:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:43:16.668802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88130ea-5638-45a0-b210-2aac33e5ad18 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:16.690199 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88130ea-5638-45a0-b210-2aac33e5ad18 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:16.703531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9671993a-c725-4c12-b6a9-1112238a9b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:16.704295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9671993a-c725-4c12-b6a9-1112238a9b55 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:16.704550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9671993a-c725-4c12-b6a9-1112238a9b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:16.704825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9671993a-c725-4c12-b6a9-1112238a9b55 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:16.707894 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9671993a-c725-4c12-b6a9-1112238a9b55 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:16.712559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9671993a-c725-4c12-b6a9-1112238a9b55 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:16.725465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:16.825089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1131/2276] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:43:16 2022] POST /identity/v3/projects => generated 373 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:43:16.831301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:16.886254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee52006-891b-486a-b9a8-f0a8a04f344c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:17.142260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963709ff-2e2e-4f98-805b-d55345f2e1a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:17.145703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963709ff-2e2e-4f98-805b-d55345f2e1a2 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:17.147468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963709ff-2e2e-4f98-805b-d55345f2e1a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:17.148067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963709ff-2e2e-4f98-805b-d55345f2e1a2 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:17.151334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963709ff-2e2e-4f98-805b-d55345f2e1a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:17.158327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963709ff-2e2e-4f98-805b-d55345f2e1a2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:17.229222 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1146/2277] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:43:16 2022] POST /identity/v3/users => generated 287 bytes in 401 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:43:17.233966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9671993a-c725-4c12-b6a9-1112238a9b55 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:17.256082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9671993a-c725-4c12-b6a9-1112238a9b55 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:17.268686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf79b08-5bd1-49bc-bfa3-9f2ebf2510a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:17.269977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf79b08-5bd1-49bc-bfa3-9f2ebf2510a2 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:17.270643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf79b08-5bd1-49bc-bfa3-9f2ebf2510a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:17.271367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf79b08-5bd1-49bc-bfa3-9f2ebf2510a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:17.272947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf79b08-5bd1-49bc-bfa3-9f2ebf2510a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:17.276777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf79b08-5bd1-49bc-bfa3-9f2ebf2510a2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:17.315194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1132/2278] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:43:17 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:43:17.324040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963709ff-2e2e-4f98-805b-d55345f2e1a2 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:17.348723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963709ff-2e2e-4f98-805b-d55345f2e1a2 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:17.402066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dceec8e4-ca69-4876-a7f3-d271d683f65f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:17.406440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceec8e4-ca69-4876-a7f3-d271d683f65f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:17.409418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceec8e4-ca69-4876-a7f3-d271d683f65f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:17.415002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceec8e4-ca69-4876-a7f3-d271d683f65f admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:17.421221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dceec8e4-ca69-4876-a7f3-d271d683f65f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:17.437696 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dceec8e4-ca69-4876-a7f3-d271d683f65f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:17.520276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1147/2279] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:43:17 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:43:17.527103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf79b08-5bd1-49bc-bfa3-9f2ebf2510a2 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:17.560993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf79b08-5bd1-49bc-bfa3-9f2ebf2510a2 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:17.583076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:17.584683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:17.586184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:17.587014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] PATH_INFO: `/v3/projects/a2e66667ce1d4a8e87dc287d74ed6991/users/2d3be77ead284ca7a85a57179e37c457/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:17.623471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2e66667ce1d4a8e87dc287d74ed6991, user_id=2d3be77ead284ca7a85a57179e37c457, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:17.627026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:17.648946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1133/2280] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:43:17 2022] PUT /identity/v3/projects/a2e66667ce1d4a8e87dc287d74ed6991/users/2d3be77ead284ca7a85a57179e37c457/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:43:17.764756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a1f582-fcca-4d4d-b835-d8417555c318 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:17.765226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a1f582-fcca-4d4d-b835-d8417555c318 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:17.765709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a1f582-fcca-4d4d-b835-d8417555c318 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:17.833347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f4a1f582-fcca-4d4d-b835-d8417555c318 None None] MFA Rules not processed for user `2d3be77ead284ca7a85a57179e37c457`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:17.836333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f4a1f582-fcca-4d4d-b835-d8417555c318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:18.034767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1148/2281] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:18.053539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:18.063183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:18.391257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321e2432-d983-4841-9d7e-02e85d17b664 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:18.412230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cef1a2-053f-451b-8597-f0aa4bd4a058 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:18.413282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cef1a2-053f-451b-8597-f0aa4bd4a058 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:18.413767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cef1a2-053f-451b-8597-f0aa4bd4a058 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:18.414212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cef1a2-053f-451b-8597-f0aa4bd4a058 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:18.440830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cef1a2-053f-451b-8597-f0aa4bd4a058 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:18.446048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cef1a2-053f-451b-8597-f0aa4bd4a058 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:18.472450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1134/2282] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:43:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:43:19.895579 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a1f582-fcca-4d4d-b835-d8417555c318 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:19.899115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f4a1f582-fcca-4d4d-b835-d8417555c318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:20.016707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:20.205564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a1f582-fcca-4d4d-b835-d8417555c318 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:20.219973 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:20.220933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:20.221821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:20.222547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] PATH_INFO: `/v3/users/a4674e80ea68459499781e922caa33a9` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:20.226025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4674e80ea68459499781e922caa33a9)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:20.228422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:20.390463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4674e80ea68459499781e922caa33a9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:20.403785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4674e80ea68459499781e922caa33a9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:20.404837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4674e80ea68459499781e922caa33a9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:20.405546 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4674e80ea68459499781e922caa33a9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:20.436937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4674e80ea68459499781e922caa33a9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:20.480699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1149/2283] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:43:19 2022] DELETE /identity/v3/users/a4674e80ea68459499781e922caa33a9 => generated 0 bytes in 586 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:43:20.487135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cdf46b-cee9-4533-a331-3d5321bf1938 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:20.488870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cdf46b-cee9-4533-a331-3d5321bf1938 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:20.489552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cdf46b-cee9-4533-a331-3d5321bf1938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:20.564880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d8cdf46b-cee9-4533-a331-3d5321bf1938 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:20.567400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d8cdf46b-cee9-4533-a331-3d5321bf1938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:20.702782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1135/2284] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:43:20.761835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:20.763377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:20.783603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b27353-b002-4137-bcf2-7350a7689d84 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:20.784067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b27353-b002-4137-bcf2-7350a7689d84 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:20.784532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b27353-b002-4137-bcf2-7350a7689d84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:20.848521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-78b27353-b002-4137-bcf2-7350a7689d84 None None] MFA Rules not processed for user `2d3be77ead284ca7a85a57179e37c457`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:20.851157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-78b27353-b002-4137-bcf2-7350a7689d84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:20.919790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6e6edb-8ee1-4dfd-8803-278d9cea626e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:20.935434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:20.939480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:20.940062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:20.940580 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:20.985834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:20.988041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:21.018938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1150/2285] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:43:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:43:21.024260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:21.044288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1136/2286] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:22.784457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:22.791956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:22.843213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2a8541-5a38-42f7-9325-e88d3030f1e5 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:22.879631 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:22.880743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:22.881230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:22.881731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] PATH_INFO: `/v3/projects/ee8671b8d3024f6aac49368289f26eb7` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:22.899757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee8671b8d3024f6aac49368289f26eb7)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:22.903583 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:23.054347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:23.097258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee8671b8d3024f6aac49368289f26eb7', 'request_id': 'req-3897ddc2-9f2f-4040-901c-1d80a50f1080'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:23.116901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee8671b8d3024f6aac49368289f26eb7', 'request_id': 'req-3897ddc2-9f2f-4040-901c-1d80a50f1080'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:23.124582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee8671b8d3024f6aac49368289f26eb7', 'request_id': 'req-3897ddc2-9f2f-4040-901c-1d80a50f1080'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:23.183411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] The token cache is being invalidate because project ee8671b8d3024f6aac49368289f26eb7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:43:23.183653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:23.184316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3897ddc2-9f2f-4040-901c-1d80a50f1080 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:43:23.185924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1151/2287] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:43:22 2022] DELETE /identity/v3/projects/ee8671b8d3024f6aac49368289f26eb7 => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:43:23.192458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c072bd76-15d8-41aa-b3b4-8578dfe3609a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:23.193114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c072bd76-15d8-41aa-b3b4-8578dfe3609a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:23.193114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c072bd76-15d8-41aa-b3b4-8578dfe3609a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:23.255774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c072bd76-15d8-41aa-b3b4-8578dfe3609a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:23.258803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c072bd76-15d8-41aa-b3b4-8578dfe3609a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:23.451696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1137/2288] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:23.460345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b304ff-6947-4933-8c71-85577967bf80 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:23.460634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b304ff-6947-4933-8c71-85577967bf80 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:23.460909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b304ff-6947-4933-8c71-85577967bf80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:23.533154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-49b304ff-6947-4933-8c71-85577967bf80 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:23.534535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-49b304ff-6947-4933-8c71-85577967bf80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:23.570071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1152/2289] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:23.577707 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c072bd76-15d8-41aa-b3b4-8578dfe3609a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:23.597546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c072bd76-15d8-41aa-b3b4-8578dfe3609a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:23.608578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:23.609669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:23.610378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:23.611010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:23.612003 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:43:23.614045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:23.616078 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:23.667218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1138/2290] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:43:23.668456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b304ff-6947-4933-8c71-85577967bf80 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:23.704411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b304ff-6947-4933-8c71-85577967bf80 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:23.714438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326672b2-dc7f-40dd-b1b6-df29020549f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:23.715721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326672b2-dc7f-40dd-b1b6-df29020549f6 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:23.717847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326672b2-dc7f-40dd-b1b6-df29020549f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:23.718486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326672b2-dc7f-40dd-b1b6-df29020549f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:23.720158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326672b2-dc7f-40dd-b1b6-df29020549f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:23.735650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326672b2-dc7f-40dd-b1b6-df29020549f6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:23.853350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1153/2291] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:43:23 2022] POST /identity/v3/projects => generated 365 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:43:23.856233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:23.873888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48bfbfd-8efd-41ea-95ce-c00940c392eb admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:24.010559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d72c7c-b14f-4389-b66f-78175f3b6cfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:24.012185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d72c7c-b14f-4389-b66f-78175f3b6cfc admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:24.012930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d72c7c-b14f-4389-b66f-78175f3b6cfc admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:24.013374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d72c7c-b14f-4389-b66f-78175f3b6cfc admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:24.014403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d72c7c-b14f-4389-b66f-78175f3b6cfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:24.017617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d72c7c-b14f-4389-b66f-78175f3b6cfc admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:24.060122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1139/2292] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:43:23 2022] POST /identity/v3/users => generated 283 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:43:24.061884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326672b2-dc7f-40dd-b1b6-df29020549f6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:24.080905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326672b2-dc7f-40dd-b1b6-df29020549f6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:24.093564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa435b1-bdc2-492c-afa5-3ce23bb0dc59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:24.098368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa435b1-bdc2-492c-afa5-3ce23bb0dc59 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:24.098919 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa435b1-bdc2-492c-afa5-3ce23bb0dc59 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:24.099888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa435b1-bdc2-492c-afa5-3ce23bb0dc59 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:24.105365 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa435b1-bdc2-492c-afa5-3ce23bb0dc59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:24.111832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa435b1-bdc2-492c-afa5-3ce23bb0dc59 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:24.172900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d72c7c-b14f-4389-b66f-78175f3b6cfc admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:24.182937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1154/2293] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:43:24 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:43:24.208703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d72c7c-b14f-4389-b66f-78175f3b6cfc admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:24.226981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37230aa6-d283-445c-9530-9f7eccd71388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:24.228063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37230aa6-d283-445c-9530-9f7eccd71388 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:24.228556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37230aa6-d283-445c-9530-9f7eccd71388 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:24.229039 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37230aa6-d283-445c-9530-9f7eccd71388 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:24.230258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37230aa6-d283-445c-9530-9f7eccd71388 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:24.233647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37230aa6-d283-445c-9530-9f7eccd71388 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:24.260873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1140/2294] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:43:24 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:43:24.268477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa435b1-bdc2-492c-afa5-3ce23bb0dc59 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:24.298350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa435b1-bdc2-492c-afa5-3ce23bb0dc59 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:24.328876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:24.330786 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:24.348897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:24.349383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] PATH_INFO: `/v3/projects/6bfffd63fa3b450da550558a86452882/users/6af2f2ca868c46298b296c651f7706e6/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:24.406368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bfffd63fa3b450da550558a86452882, user_id=6af2f2ca868c46298b296c651f7706e6, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:24.423693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:24.476573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1155/2295] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:43:24 2022] PUT /identity/v3/projects/6bfffd63fa3b450da550558a86452882/users/6af2f2ca868c46298b296c651f7706e6/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:43:24.600681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596b38aa-4839-49a8-a480-91bef45dec33 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:24.601274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596b38aa-4839-49a8-a480-91bef45dec33 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:24.601755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596b38aa-4839-49a8-a480-91bef45dec33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:24.747885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-596b38aa-4839-49a8-a480-91bef45dec33 None None] MFA Rules not processed for user `6af2f2ca868c46298b296c651f7706e6`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:24.750437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-596b38aa-4839-49a8-a480-91bef45dec33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:24.898376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:24.957479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1141/2296] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:43:24 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 359 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:25.025037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:25.028424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:25.168939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d4efcb-a166-4ab1-afce-8d49267b1fe6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:25.176077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343d0d97-cf6a-4269-abb4-3c2a224fff64 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:25.177479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343d0d97-cf6a-4269-abb4-3c2a224fff64 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:25.178026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343d0d97-cf6a-4269-abb4-3c2a224fff64 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:25.178587 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343d0d97-cf6a-4269-abb4-3c2a224fff64 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:25.193908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343d0d97-cf6a-4269-abb4-3c2a224fff64 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:25.196864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343d0d97-cf6a-4269-abb4-3c2a224fff64 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:25.218454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1156/2297] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:43:25 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:43:34.929595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:35.208951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=65,keystone:DELETE=5,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:44.809883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4ce318-818a-41d9-ab28-0688f395e72a None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:44.810512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4ce318-818a-41d9-ab28-0688f395e72a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:44.810512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4ce318-818a-41d9-ab28-0688f395e72a None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:44.811939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1142/2298] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:43:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:43:44.818711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26009a0f-5426-4227-9bf6-af99c173607b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:44.819276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26009a0f-5426-4227-9bf6-af99c173607b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:44.819276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26009a0f-5426-4227-9bf6-af99c173607b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:44.843288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-26009a0f-5426-4227-9bf6-af99c173607b None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:44.844952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-26009a0f-5426-4227-9bf6-af99c173607b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:44.928921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1157/2299] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:43:44 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:43:44.945492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4ce318-818a-41d9-ab28-0688f395e72a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:44.947473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4c4ce318-818a-41d9-ab28-0688f395e72a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:45.008024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4ce318-818a-41d9-ab28-0688f395e72a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:45.011722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cde8b4-4f8c-490a-b1fa-e2703b0342f1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:45.012484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cde8b4-4f8c-490a-b1fa-e2703b0342f1 service glance] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:45.012761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cde8b4-4f8c-490a-b1fa-e2703b0342f1 service glance] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:45.013029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cde8b4-4f8c-490a-b1fa-e2703b0342f1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:45.024729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cde8b4-4f8c-490a-b1fa-e2703b0342f1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:45.026178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cde8b4-4f8c-490a-b1fa-e2703b0342f1 service glance] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:45.040182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1143/2300] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:43:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:43:45.127242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4162a556-7ef4-4cb9-ab50-1d9aaebe19a5 None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:45.127421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4162a556-7ef4-4cb9-ab50-1d9aaebe19a5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:45.127686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4162a556-7ef4-4cb9-ab50-1d9aaebe19a5 None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:45.129067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1158/2301] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:43:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:43:45.136564 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a13e480-ebaa-499c-a900-d48fc2189a51 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:45.136950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a13e480-ebaa-499c-a900-d48fc2189a51 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:45.137372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a13e480-ebaa-499c-a900-d48fc2189a51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:45.161350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-8a13e480-ebaa-499c-a900-d48fc2189a51 None None] MFA Rules not processed for user `dea13198ef8e45de85b92e79e3ee71b0`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:45.163182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8a13e480-ebaa-499c-a900-d48fc2189a51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:45.241293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1144/2302] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:43:45 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:45.253127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4162a556-7ef4-4cb9-ab50-1d9aaebe19a5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:45.254865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4162a556-7ef4-4cb9-ab50-1d9aaebe19a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:45.268447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4162a556-7ef4-4cb9-ab50-1d9aaebe19a5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:45.272011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfcb936-c102-4059-9124-c4a0bf158507 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:45.272610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfcb936-c102-4059-9124-c4a0bf158507 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:45.272889 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfcb936-c102-4059-9124-c4a0bf158507 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:45.273162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfcb936-c102-4059-9124-c4a0bf158507 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:45.281189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfcb936-c102-4059-9124-c4a0bf158507 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:45.282626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfcb936-c102-4059-9124-c4a0bf158507 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:45.292383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1159/2303] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:43:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:43:45.386655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88aac67-69e7-4a57-849b-c23481cfb3a8 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:45.387276 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88aac67-69e7-4a57-849b-c23481cfb3a8 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:45.387276 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88aac67-69e7-4a57-849b-c23481cfb3a8 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:45.389130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1145/2304] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:43:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:43:45.395996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1d62c5-2116-48bf-aa97-daae2c5f4762 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:45.396287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1d62c5-2116-48bf-aa97-daae2c5f4762 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:45.396823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1d62c5-2116-48bf-aa97-daae2c5f4762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:45.428787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5f1d62c5-2116-48bf-aa97-daae2c5f4762 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:43:45.431006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5f1d62c5-2116-48bf-aa97-daae2c5f4762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:45.443859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1160/2305] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:43:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:43:45.456291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88aac67-69e7-4a57-849b-c23481cfb3a8 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:43:45.458154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b88aac67-69e7-4a57-849b-c23481cfb3a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:43:45.474471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88aac67-69e7-4a57-849b-c23481cfb3a8 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:43:45.478111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:43:45.478816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:43:45.479082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:43:45.479350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:43:45.503419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:43:45.509495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:43:45.532007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1146/2306] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:43:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:43:55.426164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=40 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:43:55.518885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=55 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:44:02.131986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c21df-1467-4826-9ac2-38de47109d43 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:02.132689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c21df-1467-4826-9ac2-38de47109d43 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:02.132933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c21df-1467-4826-9ac2-38de47109d43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:02.154961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-129c21df-1467-4826-9ac2-38de47109d43 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:02.156231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-129c21df-1467-4826-9ac2-38de47109d43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:02.217644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1161/2307] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:44:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:02.230568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:02.232216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:02.307526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c758e-8bc1-40a3-a972-f699e08e1d89 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:02.314905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a331b7-9847-42fa-a07e-168b16b6e652 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:02.315549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a331b7-9847-42fa-a07e-168b16b6e652 service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:02.315859 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a331b7-9847-42fa-a07e-168b16b6e652 service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:02.316251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a331b7-9847-42fa-a07e-168b16b6e652 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:02.325670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a331b7-9847-42fa-a07e-168b16b6e652 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:02.327209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a331b7-9847-42fa-a07e-168b16b6e652 service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:02.335558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1147/2308] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:44:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:44:12.210565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=25 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:44:12.332142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=32 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:44:18.044197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c21df-1467-4826-9ac2-38de47109d43 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:18.049239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-129c21df-1467-4826-9ac2-38de47109d43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:18.170831 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c21df-1467-4826-9ac2-38de47109d43 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:18.176246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d33902-9f8f-45a6-8b1b-683155102e64 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:18.176880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d33902-9f8f-45a6-8b1b-683155102e64 service placement] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:18.177168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d33902-9f8f-45a6-8b1b-683155102e64 service placement] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:18.177445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d33902-9f8f-45a6-8b1b-683155102e64 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:18.179789 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-52d33902-9f8f-45a6-8b1b-683155102e64 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:18.209023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d33902-9f8f-45a6-8b1b-683155102e64 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:18.211074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d33902-9f8f-45a6-8b1b-683155102e64 service placement] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:18.228522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1162/2309] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:44:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:44:21.384686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a331b7-9847-42fa-a07e-168b16b6e652 service swift] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:21.469500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a331b7-9847-42fa-a07e-168b16b6e652 service swift] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:21.474379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:21.475055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:21.475415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:21.475686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] PATH_INFO: `/v3/users/6af2f2ca868c46298b296c651f7706e6` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:21.476870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6af2f2ca868c46298b296c651f7706e6)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:21.478828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:21.539992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6af2f2ca868c46298b296c651f7706e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:21.540649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6af2f2ca868c46298b296c651f7706e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:21.548444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6af2f2ca868c46298b296c651f7706e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:21.556876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6af2f2ca868c46298b296c651f7706e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:21.572192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6af2f2ca868c46298b296c651f7706e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:21.579148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1148/2310] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:44:21 2022] DELETE /identity/v3/users/6af2f2ca868c46298b296c651f7706e6 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:44:21.583836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a07f84-08eb-4370-a1eb-d8c6a6bfbac9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:21.584395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a07f84-08eb-4370-a1eb-d8c6a6bfbac9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:21.584395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a07f84-08eb-4370-a1eb-d8c6a6bfbac9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:21.631382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f0a07f84-08eb-4370-a1eb-d8c6a6bfbac9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:21.634402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f0a07f84-08eb-4370-a1eb-d8c6a6bfbac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:21.738276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1163/2311] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:44:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:21.748649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:21.750097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:21.838258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b5625d-5606-40c0-8809-fc601e5c1ef4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:21.843053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4375018f-87a9-42eb-9e96-1639fff3a1ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:21.843972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375018f-87a9-42eb-9e96-1639fff3a1ed service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:21.844250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375018f-87a9-42eb-9e96-1639fff3a1ed service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:21.844525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375018f-87a9-42eb-9e96-1639fff3a1ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:21.856531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375018f-87a9-42eb-9e96-1639fff3a1ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:21.858001 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375018f-87a9-42eb-9e96-1639fff3a1ed service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:21.870866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1149/2312] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:44:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:44:22.572674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a07f84-08eb-4370-a1eb-d8c6a6bfbac9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:22.574592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f0a07f84-08eb-4370-a1eb-d8c6a6bfbac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:22.607574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a07f84-08eb-4370-a1eb-d8c6a6bfbac9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:22.623949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:22.624970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:22.625422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:22.625696 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] PATH_INFO: `/v3/projects/6bfffd63fa3b450da550558a86452882` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:22.628717 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bfffd63fa3b450da550558a86452882)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:22.632753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:22.720294 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bfffd63fa3b450da550558a86452882', 'request_id': 'req-6a3a4489-3c6e-424b-9773-cc5d4e82257b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:22.727845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bfffd63fa3b450da550558a86452882', 'request_id': 'req-6a3a4489-3c6e-424b-9773-cc5d4e82257b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:22.734363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bfffd63fa3b450da550558a86452882', 'request_id': 'req-6a3a4489-3c6e-424b-9773-cc5d4e82257b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:22.746710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] The token cache is being invalidate because project 6bfffd63fa3b450da550558a86452882 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:44:22.747025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:22.747665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6a3a4489-3c6e-424b-9773-cc5d4e82257b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:22.749449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1164/2313] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:44:22 2022] DELETE /identity/v3/projects/6bfffd63fa3b450da550558a86452882 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:44:26.369688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d5e241-d1f9-423e-a2a5-4bc2aa8cc329 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:26.373441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d5e241-d1f9-423e-a2a5-4bc2aa8cc329 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:26.373711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d5e241-d1f9-423e-a2a5-4bc2aa8cc329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:26.418444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-17d5e241-d1f9-423e-a2a5-4bc2aa8cc329 None None] MFA Rules not processed for user `8b46b3fa1df24c80ab7315c4ff44c064`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:26.419922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-17d5e241-d1f9-423e-a2a5-4bc2aa8cc329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:26.448509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:44:26.558142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1150/2314] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:44:26 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:26.641256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a4b4d5-2cdb-4e3c-a6ac-542e01d4c997 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:26.641775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a4b4d5-2cdb-4e3c-a6ac-542e01d4c997 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:26.641877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a4b4d5-2cdb-4e3c-a6ac-542e01d4c997 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:26.679339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-72a4b4d5-2cdb-4e3c-a6ac-542e01d4c997 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:26.680409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-72a4b4d5-2cdb-4e3c-a6ac-542e01d4c997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:26.696192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:44:26.799748 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1165/2315] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:44:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:26.827044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d5e241-d1f9-423e-a2a5-4bc2aa8cc329 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:26.828638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-17d5e241-d1f9-423e-a2a5-4bc2aa8cc329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:26.931953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d5e241-d1f9-423e-a2a5-4bc2aa8cc329 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:26.941472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cffc3a2-b53d-4a44-b822-3a3c07f94fbc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:26.942653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cffc3a2-b53d-4a44-b822-3a3c07f94fbc service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:26.943168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cffc3a2-b53d-4a44-b822-3a3c07f94fbc service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:26.943661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cffc3a2-b53d-4a44-b822-3a3c07f94fbc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:26.956067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cffc3a2-b53d-4a44-b822-3a3c07f94fbc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:26.958811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cffc3a2-b53d-4a44-b822-3a3c07f94fbc service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:26.971994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1151/2316] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:44:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:44:27.818010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c6662c-4e9b-4ee4-94f2-95af138374ec None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:27.819061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c6662c-4e9b-4ee4-94f2-95af138374ec None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:27.819816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c6662c-4e9b-4ee4-94f2-95af138374ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:27.860517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-03c6662c-4e9b-4ee4-94f2-95af138374ec None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:27.861778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-03c6662c-4e9b-4ee4-94f2-95af138374ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:27.882158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1166/2317] 10.222.0.77 () {58 vars in 967 bytes} [Mon Aug 29 22:44:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:27.902292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cffc3a2-b53d-4a44-b822-3a3c07f94fbc service swift] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:27.930065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cffc3a2-b53d-4a44-b822-3a3c07f94fbc service swift] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:27.936479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc549b7e-bc8c-4271-a7db-f7777dd7ed12 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:27.937277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc549b7e-bc8c-4271-a7db-f7777dd7ed12 service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:27.937559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc549b7e-bc8c-4271-a7db-f7777dd7ed12 service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:27.937939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc549b7e-bc8c-4271-a7db-f7777dd7ed12 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:27.950032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc549b7e-bc8c-4271-a7db-f7777dd7ed12 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:27.951837 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc549b7e-bc8c-4271-a7db-f7777dd7ed12 service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:27.982986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1152/2318] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:44:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:44:28.366688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ea0cd-cad5-47ee-bfdc-03b21b5f6e65 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:28.367438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ea0cd-cad5-47ee-bfdc-03b21b5f6e65 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:28.367691 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ea0cd-cad5-47ee-bfdc-03b21b5f6e65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:28.391228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-dc9ea0cd-cad5-47ee-bfdc-03b21b5f6e65 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:28.393964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dc9ea0cd-cad5-47ee-bfdc-03b21b5f6e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:28.401888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1167/2319] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:44:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:44:28.414699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc549b7e-bc8c-4271-a7db-f7777dd7ed12 service swift] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:28.429854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc549b7e-bc8c-4271-a7db-f7777dd7ed12 service swift] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:28.439057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec51a75a-4364-4a98-a217-9b634f35de66 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:28.440191 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec51a75a-4364-4a98-a217-9b634f35de66 service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:28.440690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec51a75a-4364-4a98-a217-9b634f35de66 service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:28.441184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec51a75a-4364-4a98-a217-9b634f35de66 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:28.457217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec51a75a-4364-4a98-a217-9b634f35de66 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:28.460846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec51a75a-4364-4a98-a217-9b634f35de66 service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:28.477193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1153/2320] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:44:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:44:28.843683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdcdb11-fb55-4ade-bad0-61a239568d3c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:28.844158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdcdb11-fb55-4ade-bad0-61a239568d3c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:28.844252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdcdb11-fb55-4ade-bad0-61a239568d3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:28.870445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-dcdcdb11-fb55-4ade-bad0-61a239568d3c None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:28.870445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dcdcdb11-fb55-4ade-bad0-61a239568d3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:28.873430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1168/2321] 10.222.0.77 () {58 vars in 967 bytes} [Mon Aug 29 22:44:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:28.887849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec51a75a-4364-4a98-a217-9b634f35de66 service swift] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:28.901159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec51a75a-4364-4a98-a217-9b634f35de66 service swift] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:28.907119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4320bc20-054d-4d2e-a1d4-17162187d9a5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:28.908171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320bc20-054d-4d2e-a1d4-17162187d9a5 service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:28.908522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320bc20-054d-4d2e-a1d4-17162187d9a5 service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:28.909014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320bc20-054d-4d2e-a1d4-17162187d9a5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:28.919510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4320bc20-054d-4d2e-a1d4-17162187d9a5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:28.921116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4320bc20-054d-4d2e-a1d4-17162187d9a5 service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:28.933424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1154/2322] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:44:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:44:31.417421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdcdb11-fb55-4ade-bad0-61a239568d3c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:31.418998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dcdcdb11-fb55-4ade-bad0-61a239568d3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:31.527677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdcdb11-fb55-4ade-bad0-61a239568d3c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:31.532425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:31.533095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:31.533382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:31.533656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] PATH_INFO: `/v3/users/2d3be77ead284ca7a85a57179e37c457` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:31.534918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d3be77ead284ca7a85a57179e37c457)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:31.536869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:31.596396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d3be77ead284ca7a85a57179e37c457'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:31.604876 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d3be77ead284ca7a85a57179e37c457'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:31.605612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d3be77ead284ca7a85a57179e37c457'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:31.606309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d3be77ead284ca7a85a57179e37c457'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:31.615288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d3be77ead284ca7a85a57179e37c457'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:31.645732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1169/2323] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:44:31 2022] DELETE /identity/v3/users/2d3be77ead284ca7a85a57179e37c457 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:44:31.650817 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0449c9-3302-43ce-9858-c476c6cda431 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:31.651306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0449c9-3302-43ce-9858-c476c6cda431 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:31.651424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0449c9-3302-43ce-9858-c476c6cda431 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:31.672193 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-be0449c9-3302-43ce-9858-c476c6cda431 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:31.673435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-be0449c9-3302-43ce-9858-c476c6cda431 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:31.764248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1155/2324] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:44:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:31.774866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:31.776387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:31.798621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:44:31.862644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1ebbb-0428-4a55-a896-724f07a1fa10 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:31.871021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd98d157-d481-45ea-b984-acee318f6930 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:31.872019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd98d157-d481-45ea-b984-acee318f6930 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:31.872327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd98d157-d481-45ea-b984-acee318f6930 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:31.872798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd98d157-d481-45ea-b984-acee318f6930 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:31.884109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd98d157-d481-45ea-b984-acee318f6930 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:31.885804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd98d157-d481-45ea-b984-acee318f6930 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:31.898267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1170/2325] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:44:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:44:32.222153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0449c9-3302-43ce-9858-c476c6cda431 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:32.224594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-be0449c9-3302-43ce-9858-c476c6cda431 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:32.242363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0449c9-3302-43ce-9858-c476c6cda431 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:32.249564 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:32.250275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:32.250565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:32.250884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] PATH_INFO: `/v3/projects/a2e66667ce1d4a8e87dc287d74ed6991` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:32.252629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2e66667ce1d4a8e87dc287d74ed6991)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:32.255403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:32.299621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:44:32.358352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2e66667ce1d4a8e87dc287d74ed6991', 'request_id': 'req-063b34f3-9cce-48a9-9dfc-da59ce61efa0'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:32.364529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2e66667ce1d4a8e87dc287d74ed6991', 'request_id': 'req-063b34f3-9cce-48a9-9dfc-da59ce61efa0'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:32.373752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2e66667ce1d4a8e87dc287d74ed6991', 'request_id': 'req-063b34f3-9cce-48a9-9dfc-da59ce61efa0'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:32.385479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] The token cache is being invalidate because project a2e66667ce1d4a8e87dc287d74ed6991 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:44:32.385768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:32.386396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-063b34f3-9cce-48a9-9dfc-da59ce61efa0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:44:32.387903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1156/2326] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:44:32 2022] DELETE /identity/v3/projects/a2e66667ce1d4a8e87dc287d74ed6991 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:44:35.103685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394c937-7bf6-4767-8007-08b9e7382f8e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:35.104228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394c937-7bf6-4767-8007-08b9e7382f8e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:35.104228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394c937-7bf6-4767-8007-08b9e7382f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:35.129160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-6394c937-7bf6-4767-8007-08b9e7382f8e None None] MFA Rules not processed for user `2d0e45a9ee9644f8b9eac57d9a14d14e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:35.130574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6394c937-7bf6-4767-8007-08b9e7382f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:35.197861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1171/2327] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:44:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:44:36.962911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0acaecd-0180-4fb6-9751-13b69fcca262 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:36.963886 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0acaecd-0180-4fb6-9751-13b69fcca262 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:36.964390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0acaecd-0180-4fb6-9751-13b69fcca262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:36.989847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d0acaecd-0180-4fb6-9751-13b69fcca262 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:36.991468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d0acaecd-0180-4fb6-9751-13b69fcca262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:37.077724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1157/2328] 10.222.0.77 () {58 vars in 967 bytes} [Mon Aug 29 22:44:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:37.083658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6394c937-7bf6-4767-8007-08b9e7382f8e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:44:37.085715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6394c937-7bf6-4767-8007-08b9e7382f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:37.177339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6394c937-7bf6-4767-8007-08b9e7382f8e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:44:37.184004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62ca7f-04b3-4915-9178-cb5e69209319 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:44:37.184719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62ca7f-04b3-4915-9178-cb5e69209319 service swift] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:37.185179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62ca7f-04b3-4915-9178-cb5e69209319 service swift] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:37.185593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62ca7f-04b3-4915-9178-cb5e69209319 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:37.194837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62ca7f-04b3-4915-9178-cb5e69209319 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:44:37.196805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62ca7f-04b3-4915-9178-cb5e69209319 service swift] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:44:37.207559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1172/2329] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:44:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:44:42.146207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5432fa9-55cb-4867-875b-9390df63162c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:44:42.148069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5432fa9-55cb-4867-875b-9390df63162c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:44:42.148695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5432fa9-55cb-4867-875b-9390df63162c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:44:42.189654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c5432fa9-55cb-4867-875b-9390df63162c None None] MFA Rules not processed for user `cc7aa14777ab43a083b3a120c4349d67`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:44:42.191346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c5432fa9-55cb-4867-875b-9390df63162c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:44:42.267988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1158/2330] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:44:42 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:44:47.204069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=75 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:44:52.259652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=69,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:45:36.300632 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cfc3eb-abbd-4b8e-b6bb-717d6445359a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:45:36.301243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cfc3eb-abbd-4b8e-b6bb-717d6445359a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:45:36.301243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cfc3eb-abbd-4b8e-b6bb-717d6445359a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:45:36.322957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f5cfc3eb-abbd-4b8e-b6bb-717d6445359a None None] MFA Rules not processed for user `003dd6612cda48618966db8c0788b794`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:45:36.324196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f5cfc3eb-abbd-4b8e-b6bb-717d6445359a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:45:36.384651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1173/2331] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:45:36 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:45:38.747754 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5432fa9-55cb-4867-875b-9390df63162c None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:45:38.749750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c5432fa9-55cb-4867-875b-9390df63162c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:45:38.829986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5432fa9-55cb-4867-875b-9390df63162c None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:45:38.834777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:45:38.835431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:45:38.835708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:45:38.835980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] PATH_INFO: `/v3/users/2d0e45a9ee9644f8b9eac57d9a14d14e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:45:38.837466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d0e45a9ee9644f8b9eac57d9a14d14e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:45:38.839481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:45:38.895154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d0e45a9ee9644f8b9eac57d9a14d14e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:38.896037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d0e45a9ee9644f8b9eac57d9a14d14e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:38.904251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d0e45a9ee9644f8b9eac57d9a14d14e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:38.913396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d0e45a9ee9644f8b9eac57d9a14d14e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:38.928269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1a3ba5a2-b3dd-43d2-b110-09d089527e13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d0e45a9ee9644f8b9eac57d9a14d14e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:38.935475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1159/2332] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:45:38 2022] DELETE /identity/v3/users/2d0e45a9ee9644f8b9eac57d9a14d14e => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:45:39.202997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cfc3eb-abbd-4b8e-b6bb-717d6445359a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:45:39.214848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cfc3eb-abbd-4b8e-b6bb-717d6445359a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:45:39.281228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:45:39.281881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:45:39.282166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:45:39.282461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] PATH_INFO: `/v3/projects/211213d0c20641adb48a3d39703f7532` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:45:39.283749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] RBAC: Authorizing `identity:delete_project(project_id=211213d0c20641adb48a3d39703f7532)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:45:39.286045 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:45:39.342532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '211213d0c20641adb48a3d39703f7532', 'request_id': 'req-a4dfceda-a5d3-4083-8736-d8736170a744'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:39.350421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '211213d0c20641adb48a3d39703f7532', 'request_id': 'req-a4dfceda-a5d3-4083-8736-d8736170a744'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:39.358013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '211213d0c20641adb48a3d39703f7532', 'request_id': 'req-a4dfceda-a5d3-4083-8736-d8736170a744'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:39.373362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] The token cache is being invalidate because project 211213d0c20641adb48a3d39703f7532 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:45:39.373655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:39.374318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:45:39.375915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1174/2333] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:45:39 2022] DELETE /identity/v3/projects/211213d0c20641adb48a3d39703f7532 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:45:46.105723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24788106-cfbd-4275-9010-77ab119cc6a5 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:45:46.106501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24788106-cfbd-4275-9010-77ab119cc6a5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:45:46.106501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24788106-cfbd-4275-9010-77ab119cc6a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:45:46.137168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-24788106-cfbd-4275-9010-77ab119cc6a5 None None] MFA Rules not processed for user `70da750fbe764462b50dc0fcef3eb0e6`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:45:46.139289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-24788106-cfbd-4275-9010-77ab119cc6a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:45:46.213775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1160/2334] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:45:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:45:46.279476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:45:46.281706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:45:46.342233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:45:46.381249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dfceda-a5d3-4083-8736-d8736170a744 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:45:46.386490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c06032-d5d3-4746-acd7-57d2bade53f9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:45:46.387335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c06032-d5d3-4746-acd7-57d2bade53f9 service glance] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:45:46.387680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c06032-d5d3-4746-acd7-57d2bade53f9 service glance] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:45:46.387950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c06032-d5d3-4746-acd7-57d2bade53f9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:45:46.390487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b0c06032-d5d3-4746-acd7-57d2bade53f9 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:45:46.545609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24788106-cfbd-4275-9010-77ab119cc6a5 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:45:46.547564 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-24788106-cfbd-4275-9010-77ab119cc6a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:45:46.554639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c06032-d5d3-4746-acd7-57d2bade53f9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:45:46.557314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c06032-d5d3-4746-acd7-57d2bade53f9 service glance] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:45:46.581647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1175/2335] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:45:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:45:46.598944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:45:46.660422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24788106-cfbd-4275-9010-77ab119cc6a5 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:45:46.666492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:45:46.667543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:45:46.668004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:45:46.668451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:45:46.671455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:45:46.687358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:45:46.688987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:45:46.702796 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1161/2336] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:45:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:45:56.569880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=38 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:45:56.696902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=15 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:46:05.372714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa313d5-20b2-4fec-8719-53f36454f7db None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:05.373235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa313d5-20b2-4fec-8719-53f36454f7db None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:05.373649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa313d5-20b2-4fec-8719-53f36454f7db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:05.394499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5aa313d5-20b2-4fec-8719-53f36454f7db None None] MFA Rules not processed for user `cc7aa14777ab43a083b3a120c4349d67`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:46:05.395762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5aa313d5-20b2-4fec-8719-53f36454f7db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:05.454661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1176/2337] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:46:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:46:05.466657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:46:05.468394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:05.483263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fb26df-0965-4b8a-9544-87a98f0f1dc6 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:46:05.489470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:46:05.490109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:05.490390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:05.490667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:05.492428 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:05.504513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:46:05.506122 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:46:05.516507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1162/2338] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:46:05 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:46:06.242004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa313d5-20b2-4fec-8719-53f36454f7db None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:46:06.244292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5aa313d5-20b2-4fec-8719-53f36454f7db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:06.364204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa313d5-20b2-4fec-8719-53f36454f7db None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:46:06.375400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742e2510-43bd-409c-a485-8fe5121c6624 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:46:06.376456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e2510-43bd-409c-a485-8fe5121c6624 service placement] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:06.376960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e2510-43bd-409c-a485-8fe5121c6624 service placement] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:06.377456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e2510-43bd-409c-a485-8fe5121c6624 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:06.387676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-742e2510-43bd-409c-a485-8fe5121c6624 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:06.412729 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742e2510-43bd-409c-a485-8fe5121c6624 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:46:06.414493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742e2510-43bd-409c-a485-8fe5121c6624 service placement] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:46:06.433039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1177/2339] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:46:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:46:08.770423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:46:08.772201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:08.791703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5c0e4-c9e0-4c0f-8f7d-7c88875d4b3b service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:46:08.797325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:46:08.798192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:08.798698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:08.799388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:08.802000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:08.818580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:46:08.822506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:46:08.835477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1163/2340] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:46:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:46:16.424493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=57 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:46:16.594284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd043b-4200-45be-ae9b-bddf44b6a86a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:16.594787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd043b-4200-45be-ae9b-bddf44b6a86a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:16.595239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd043b-4200-45be-ae9b-bddf44b6a86a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:16.629164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-42bd043b-4200-45be-ae9b-bddf44b6a86a None None] MFA Rules not processed for user `8b46b3fa1df24c80ab7315c4ff44c064`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:46:16.631310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-42bd043b-4200-45be-ae9b-bddf44b6a86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:16.719763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1178/2341] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:46:16 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:46:18.828767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=12 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:46:19.152515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:46:19.154363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:19.183609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514f8df-1406-48ee-b43d-1c3eb60d139a service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:46:19.194245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:46:19.195460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:19.196034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:19.196631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:19.199790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:19.225976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:46:19.229174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:46:19.250936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1164/2342] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:46:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:46:26.711331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=25 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:46:28.187381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bd043b-4200-45be-ae9b-bddf44b6a86a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:46:28.210354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bd043b-4200-45be-ae9b-bddf44b6a86a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:46:28.221554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:46:28.222622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:28.223168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:28.223655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:28.226957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:28.312902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:46:28.315571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:46:28.331261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1179/2343] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:46:28 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:46:29.241470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=6 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:46:38.323095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=28 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:46:47.191810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:46:47.204970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c43d44-c6b6-4a5c-98cd-61f92bd759f4 service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:46:47.211927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315d9cc4-1960-4f9c-b4c7-5dae821573bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:46:47.212503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d9cc4-1960-4f9c-b4c7-5dae821573bc service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:47.212828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d9cc4-1960-4f9c-b4c7-5dae821573bc service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:47.213150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d9cc4-1960-4f9c-b4c7-5dae821573bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:47.215231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-315d9cc4-1960-4f9c-b4c7-5dae821573bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:47.228991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315d9cc4-1960-4f9c-b4c7-5dae821573bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:46:47.230769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315d9cc4-1960-4f9c-b4c7-5dae821573bc service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:46:47.241113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1165/2344] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:46:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:46:54.896827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:46:54.909097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609b6a88-879b-47cb-9dbe-dbae72b84dba service nova] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:46:54.916155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:46:54.917182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:54.917525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:54.917802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:54.919998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:54.943955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:46:54.947202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:46:54.970983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1180/2345] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:46:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:46:55.960905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efc94a2-8ad1-44a1-8b65-af079c4949a0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:55.962456 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efc94a2-8ad1-44a1-8b65-af079c4949a0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:55.963026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efc94a2-8ad1-44a1-8b65-af079c4949a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:56.000698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2efc94a2-8ad1-44a1-8b65-af079c4949a0 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:46:56.002982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2efc94a2-8ad1-44a1-8b65-af079c4949a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:56.083347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1166/2346] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:46:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:46:56.155564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:46:56.158721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:46:56.250823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f499fb3-437a-4bae-bd1f-746601aa816e service nova] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:46:56.260916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95455b68-f674-473c-9356-abb4bb4c8c7e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:46:56.261956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95455b68-f674-473c-9356-abb4bb4c8c7e service swift] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:46:56.262424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95455b68-f674-473c-9356-abb4bb4c8c7e service swift] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:46:56.263037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95455b68-f674-473c-9356-abb4bb4c8c7e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:46:56.274343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95455b68-f674-473c-9356-abb4bb4c8c7e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:46:56.276118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95455b68-f674-473c-9356-abb4bb4c8c7e service swift] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:46:56.286259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1181/2347] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:46:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:47:06.073017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=31 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:06.281994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=38 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:14.636533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efc94a2-8ad1-44a1-8b65-af079c4949a0 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:14.664245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efc94a2-8ad1-44a1-8b65-af079c4949a0 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:14.717421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e44e916-4aa8-44fb-8cc2-683ab631787d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:14.720284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e44e916-4aa8-44fb-8cc2-683ab631787d service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:14.720984 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e44e916-4aa8-44fb-8cc2-683ab631787d service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:14.721651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e44e916-4aa8-44fb-8cc2-683ab631787d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:14.730097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1e44e916-4aa8-44fb-8cc2-683ab631787d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:14.770278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e44e916-4aa8-44fb-8cc2-683ab631787d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:14.781083 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e44e916-4aa8-44fb-8cc2-683ab631787d service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:14.822224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1167/2348] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:16.884458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48728223-20e8-4109-b9ee-8561ac2f10ee None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:16.885256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48728223-20e8-4109-b9ee-8561ac2f10ee None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:16.885256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48728223-20e8-4109-b9ee-8561ac2f10ee None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:16.887484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1182/2349] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:47:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:47:16.896578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1848c146-4a67-4fd9-8888-e678a7c00cf1 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:16.898069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1848c146-4a67-4fd9-8888-e678a7c00cf1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:16.898867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1848c146-4a67-4fd9-8888-e678a7c00cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:16.933522 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1848c146-4a67-4fd9-8888-e678a7c00cf1 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:16.935515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1848c146-4a67-4fd9-8888-e678a7c00cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:16.953431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1168/2350] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:47:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:16.978614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48728223-20e8-4109-b9ee-8561ac2f10ee None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:17.001413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48728223-20e8-4109-b9ee-8561ac2f10ee None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:17.009265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fe8687-5817-4721-9c90-46e17a46af89 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:17.011287 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fe8687-5817-4721-9c90-46e17a46af89 service glance] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:17.012283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fe8687-5817-4721-9c90-46e17a46af89 service glance] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:17.013040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fe8687-5817-4721-9c90-46e17a46af89 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:17.041509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fe8687-5817-4721-9c90-46e17a46af89 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:17.046171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fe8687-5817-4721-9c90-46e17a46af89 service glance] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:17.086059 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1183/2351] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:17.269252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41137969-77d2-4838-b1a7-6e052b604159 None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:17.269892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41137969-77d2-4838-b1a7-6e052b604159 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:17.270785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41137969-77d2-4838-b1a7-6e052b604159 None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:17.273467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1169/2352] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:47:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:47:17.300921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19022215-d5f4-4723-b52f-6590160129b1 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:17.301194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19022215-d5f4-4723-b52f-6590160129b1 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:17.301534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19022215-d5f4-4723-b52f-6590160129b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:17.352100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-19022215-d5f4-4723-b52f-6590160129b1 None None] MFA Rules not processed for user `dea13198ef8e45de85b92e79e3ee71b0`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:17.354273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-19022215-d5f4-4723-b52f-6590160129b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:17.363787 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1184/2353] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:47:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:17.396907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41137969-77d2-4838-b1a7-6e052b604159 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:17.422836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41137969-77d2-4838-b1a7-6e052b604159 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:17.432490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58c505c-b127-4413-8cca-62ab087fecff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:17.436330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58c505c-b127-4413-8cca-62ab087fecff service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:17.438047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58c505c-b127-4413-8cca-62ab087fecff service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:17.440437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58c505c-b127-4413-8cca-62ab087fecff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:17.466700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58c505c-b127-4413-8cca-62ab087fecff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:17.487913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58c505c-b127-4413-8cca-62ab087fecff service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:17.502129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1170/2354] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:17.626146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8146571-bd83-4a3a-9168-9b8a00a0a03e None None] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:17.626625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8146571-bd83-4a3a-9168-9b8a00a0a03e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:17.626935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8146571-bd83-4a3a-9168-9b8a00a0a03e None None] PATH_INFO: `/` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:17.629241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1185/2355] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:47:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:47:17.643208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27723074-aa0c-42f9-bf59-55aaa6031a84 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:17.643844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27723074-aa0c-42f9-bf59-55aaa6031a84 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:17.644646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27723074-aa0c-42f9-bf59-55aaa6031a84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:17.683699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-27723074-aa0c-42f9-bf59-55aaa6031a84 None None] MFA Rules not processed for user `57491e4de6c3435e9d818266a910b831`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:17.685118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-27723074-aa0c-42f9-bf59-55aaa6031a84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:17.700037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1171/2356] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:47:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:17.715143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8146571-bd83-4a3a-9168-9b8a00a0a03e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:17.737119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8146571-bd83-4a3a-9168-9b8a00a0a03e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:17.743047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6969429-df28-4c2d-a730-297668d38048 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:17.744539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6969429-df28-4c2d-a730-297668d38048 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:17.745158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6969429-df28-4c2d-a730-297668d38048 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:17.746055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6969429-df28-4c2d-a730-297668d38048 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:17.775256 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6969429-df28-4c2d-a730-297668d38048 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:17.777728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6969429-df28-4c2d-a730-297668d38048 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:17.797146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1186/2357] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:25.265566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819edc92-8439-4334-a68c-11033e6eab9e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:25.267201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819edc92-8439-4334-a68c-11033e6eab9e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:25.268036 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819edc92-8439-4334-a68c-11033e6eab9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:25.305118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-819edc92-8439-4334-a68c-11033e6eab9e None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:25.307087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-819edc92-8439-4334-a68c-11033e6eab9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:25.320223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1172/2358] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:47:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:25.338350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6969429-df28-4c2d-a730-297668d38048 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:25.365387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6969429-df28-4c2d-a730-297668d38048 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:25.380383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63578bfc-3763-4cee-be5a-628d14974fa1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:25.382657 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63578bfc-3763-4cee-be5a-628d14974fa1 service swift] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:25.383415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63578bfc-3763-4cee-be5a-628d14974fa1 service swift] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:25.384027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63578bfc-3763-4cee-be5a-628d14974fa1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:25.401449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63578bfc-3763-4cee-be5a-628d14974fa1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:25.403673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63578bfc-3763-4cee-be5a-628d14974fa1 service swift] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:25.417762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1187/2359] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:47:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:47:25.464549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819edc92-8439-4334-a68c-11033e6eab9e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:25.528014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819edc92-8439-4334-a68c-11033e6eab9e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:25.564007 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:25.564860 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:25.564992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:25.565207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:25.574531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:25.624273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:25.626676 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:25.666046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1173/2360] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:25 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:31.735989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63578bfc-3763-4cee-be5a-628d14974fa1 service swift] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:31.750445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63578bfc-3763-4cee-be5a-628d14974fa1 service swift] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:31.758391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:31.759592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:31.760153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:31.760618 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:31.763697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:31.878329 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:31.880054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:31.892150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1188/2361] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:35.648197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=27 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:38.568212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:38.569760 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:38.595448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f45a8ad-0f42-4c1d-9952-c4601178429a service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:38.605703 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:38.607198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:38.607948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:38.608671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:38.619269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:38.649917 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:38.653394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:38.674795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:38.678368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:38.698466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014c7114-6277-4ed7-ab2f-67dd7257930f service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:38.698941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1174/2362] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:38.713171 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:38.714630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:38.715301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:38.715855 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] PATH_INFO: `/v3/users/cc7aa14777ab43a083b3a120c4349d67` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:38.729636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc7aa14777ab43a083b3a120c4349d67)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:38.736117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:38.918278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc7aa14777ab43a083b3a120c4349d67'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:38.934128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc7aa14777ab43a083b3a120c4349d67'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:38.935404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc7aa14777ab43a083b3a120c4349d67'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:38.936504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc7aa14777ab43a083b3a120c4349d67'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:38.951362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b584e4ff-3716-4017-8f44-6fa0625e72af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc7aa14777ab43a083b3a120c4349d67'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:39.040165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1189/2363] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:38 2022] DELETE /identity/v3/users/cc7aa14777ab43a083b3a120c4349d67 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:39.629577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:39.676120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a934944e-9d6a-46ac-8adf-0c15a174d7b2 service placement] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:39.874435 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:39.876056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:39.877011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:39.878939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] PATH_INFO: `/v3/projects/1fcb391184b94306898e43379094e5f6` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:39.883277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fcb391184b94306898e43379094e5f6)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:39.898418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:40.041531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fcb391184b94306898e43379094e5f6', 'request_id': 'req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:40.054969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fcb391184b94306898e43379094e5f6', 'request_id': 'req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:40.073282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fcb391184b94306898e43379094e5f6', 'request_id': 'req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:40.108445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] The token cache is being invalidate because project 1fcb391184b94306898e43379094e5f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:47:40.108962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:40.110602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bdab8648-7b98-4d01-9ed2-c2acfd8c4bb2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:40.118801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1175/2364] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:47:39 2022] DELETE /identity/v3/projects/1fcb391184b94306898e43379094e5f6 => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:40.123910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb4450-ec55-41dd-bdd4-04e0d193debf None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:40.125170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb4450-ec55-41dd-bdd4-04e0d193debf None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:40.125170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb4450-ec55-41dd-bdd4-04e0d193debf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:40.165061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5abb4450-ec55-41dd-bdd4-04e0d193debf None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:40.167427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5abb4450-ec55-41dd-bdd4-04e0d193debf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:40.197903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:40.331766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1190/2365] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:40.347659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba9fe7c-13e4-4e5e-9ff1-c174310e3886 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:40.348433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba9fe7c-13e4-4e5e-9ff1-c174310e3886 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:40.349175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba9fe7c-13e4-4e5e-9ff1-c174310e3886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:40.423467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-dba9fe7c-13e4-4e5e-9ff1-c174310e3886 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:40.425422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-dba9fe7c-13e4-4e5e-9ff1-c174310e3886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:40.447726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1176/2366] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:40.449596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abb4450-ec55-41dd-bdd4-04e0d193debf None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:40.470690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abb4450-ec55-41dd-bdd4-04e0d193debf None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:40.495867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:40.496893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:40.498471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:40.499362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:40.501680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:47:40.502286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:40.505275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:40.544157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1191/2367] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:47:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:47:40.560584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba9fe7c-13e4-4e5e-9ff1-c174310e3886 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:40.575346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba9fe7c-13e4-4e5e-9ff1-c174310e3886 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:40.582256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a47eec-25b8-492e-9df3-25ae6433fc6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:40.583967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a47eec-25b8-492e-9df3-25ae6433fc6c admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:40.583967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a47eec-25b8-492e-9df3-25ae6433fc6c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:40.583967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a47eec-25b8-492e-9df3-25ae6433fc6c admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:40.584841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a47eec-25b8-492e-9df3-25ae6433fc6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:40.586055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a47eec-25b8-492e-9df3-25ae6433fc6c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:40.627477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1177/2368] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:40 2022] POST /identity/v3/projects => generated 375 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:40.633864 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:40.658628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9c3abc-971e-417c-a9f6-3e8c8ee88a06 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:40.771624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056daccd-1ce8-4edf-9623-29d8893a4805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:40.772394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056daccd-1ce8-4edf-9623-29d8893a4805 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:40.772904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056daccd-1ce8-4edf-9623-29d8893a4805 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:40.773339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056daccd-1ce8-4edf-9623-29d8893a4805 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:40.774486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056daccd-1ce8-4edf-9623-29d8893a4805 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:40.778267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056daccd-1ce8-4edf-9623-29d8893a4805 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:40.813082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1192/2369] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:40 2022] POST /identity/v3/users => generated 288 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:40.818600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a47eec-25b8-492e-9df3-25ae6433fc6c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:40.837599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a47eec-25b8-492e-9df3-25ae6433fc6c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:40.846521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ff8f38-214f-4241-b58b-a228b86520c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:40.847662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff8f38-214f-4241-b58b-a228b86520c2 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:40.848168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff8f38-214f-4241-b58b-a228b86520c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:40.848660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ff8f38-214f-4241-b58b-a228b86520c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:40.849862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ff8f38-214f-4241-b58b-a228b86520c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:40.853047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ff8f38-214f-4241-b58b-a228b86520c2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:40.867932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1178/2370] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:40 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:40.874383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056daccd-1ce8-4edf-9623-29d8893a4805 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:40.892648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056daccd-1ce8-4edf-9623-29d8893a4805 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:40.902539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468bf01e-6045-4c4e-9100-835d7e6e6276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:40.902539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468bf01e-6045-4c4e-9100-835d7e6e6276 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:40.903281 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468bf01e-6045-4c4e-9100-835d7e6e6276 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:40.903858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468bf01e-6045-4c4e-9100-835d7e6e6276 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:40.905236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468bf01e-6045-4c4e-9100-835d7e6e6276 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:40.908621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468bf01e-6045-4c4e-9100-835d7e6e6276 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:40.927304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1193/2371] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:40 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:40.936679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ff8f38-214f-4241-b58b-a228b86520c2 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:40.956350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ff8f38-214f-4241-b58b-a228b86520c2 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:40.967158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:40.969075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:40.970065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:40.970519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] PATH_INFO: `/v3/projects/e366a29135794c3c8d3a2a36d07c6bfb/users/e6c282c2cf3d4521b3a49d812d48e402/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:40.997041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e366a29135794c3c8d3a2a36d07c6bfb, user_id=e6c282c2cf3d4521b3a49d812d48e402, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:41.000098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:41.019089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1179/2372] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:40 2022] PUT /identity/v3/projects/e366a29135794c3c8d3a2a36d07c6bfb/users/e6c282c2cf3d4521b3a49d812d48e402/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:41.038645 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d32a50-4172-431d-9050-9c643e69e05b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:41.039151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d32a50-4172-431d-9050-9c643e69e05b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:41.039608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d32a50-4172-431d-9050-9c643e69e05b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:41.092444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-46d32a50-4172-431d-9050-9c643e69e05b None None] MFA Rules not processed for user `e6c282c2cf3d4521b3a49d812d48e402`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:41.095123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-46d32a50-4172-431d-9050-9c643e69e05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:41.254095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1194/2373] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:41 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:41.272647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:41.273628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d32a50-4172-431d-9050-9c643e69e05b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:41.273628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-46d32a50-4172-431d-9050-9c643e69e05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:41.276263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:41.299325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:41.318150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:41.381635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba722a0-015d-484c-b8cc-cacf10dbe194 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:41.388655 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d904b072-6b03-4eca-9981-6bcd4825152e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:41.389464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d904b072-6b03-4eca-9981-6bcd4825152e service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:41.390889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d904b072-6b03-4eca-9981-6bcd4825152e service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:41.391366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d904b072-6b03-4eca-9981-6bcd4825152e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:41.394444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d904b072-6b03-4eca-9981-6bcd4825152e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:41.429941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d32a50-4172-431d-9050-9c643e69e05b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:41.445941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339e68bc-ffca-4121-a2cd-256deebe3d3c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:41.446760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339e68bc-ffca-4121-a2cd-256deebe3d3c service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:41.447266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339e68bc-ffca-4121-a2cd-256deebe3d3c service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:41.447708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339e68bc-ffca-4121-a2cd-256deebe3d3c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:41.459458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339e68bc-ffca-4121-a2cd-256deebe3d3c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:41.461238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339e68bc-ffca-4121-a2cd-256deebe3d3c service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:41.474426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1195/2374] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:41 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:41.517672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d904b072-6b03-4eca-9981-6bcd4825152e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:41.520635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d904b072-6b03-4eca-9981-6bcd4825152e service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:41.543819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1180/2375] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:41.773700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339e68bc-ffca-4121-a2cd-256deebe3d3c service nova] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:41.802145 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339e68bc-ffca-4121-a2cd-256deebe3d3c service nova] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:41.811927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:41.814910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:41.815422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:41.815740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] PATH_INFO: `/v3/users/e6c282c2cf3d4521b3a49d812d48e402` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:41.817335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6c282c2cf3d4521b3a49d812d48e402)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:41.829590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:41.932979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6c282c2cf3d4521b3a49d812d48e402'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:41.943984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6c282c2cf3d4521b3a49d812d48e402'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:41.946061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6c282c2cf3d4521b3a49d812d48e402'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:41.948482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6c282c2cf3d4521b3a49d812d48e402'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:41.964530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6c282c2cf3d4521b3a49d812d48e402'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:41.994988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1196/2376] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:41 2022] DELETE /identity/v3/users/e6c282c2cf3d4521b3a49d812d48e402 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:42.002924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea49fcfa-5b41-46c6-873e-e144b7f6afcd None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:42.003587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea49fcfa-5b41-46c6-873e-e144b7f6afcd None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:42.003830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea49fcfa-5b41-46c6-873e-e144b7f6afcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:42.039915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-ea49fcfa-5b41-46c6-873e-e144b7f6afcd None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:42.042110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ea49fcfa-5b41-46c6-873e-e144b7f6afcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:42.134455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1181/2377] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:42.159202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:42.161695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:42.271432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c929fe51-36d8-481f-bc8f-9e53cfb73137 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:42.278685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:42.279672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:42.280211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:42.280661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:42.293377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:42.295542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:42.317733 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1197/2378] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:42.607917 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe720d8-a234-42e2-94a9-dafa7e738fa2 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:42.608260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe720d8-a234-42e2-94a9-dafa7e738fa2 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:42.608565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe720d8-a234-42e2-94a9-dafa7e738fa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:42.636309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2fe720d8-a234-42e2-94a9-dafa7e738fa2 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:42.637737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2fe720d8-a234-42e2-94a9-dafa7e738fa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:42.734129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1182/2379] 10.222.0.77 () {58 vars in 967 bytes} [Mon Aug 29 22:47:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:42.761800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:42.764537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:42.801623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:42.875356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d679ce8-2930-4630-88d7-0bf5bed79d74 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:42.882545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ddb4ed-7b91-4327-b8e0-73f04f24f1f2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:42.883234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddb4ed-7b91-4327-b8e0-73f04f24f1f2 service swift] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:42.883662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddb4ed-7b91-4327-b8e0-73f04f24f1f2 service swift] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:42.883950 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddb4ed-7b91-4327-b8e0-73f04f24f1f2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:42.894070 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ddb4ed-7b91-4327-b8e0-73f04f24f1f2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:42.896195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ddb4ed-7b91-4327-b8e0-73f04f24f1f2 service swift] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:42.910423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1198/2380] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:47:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:47:43.152474 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe720d8-a234-42e2-94a9-dafa7e738fa2 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:43.154836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2fe720d8-a234-42e2-94a9-dafa7e738fa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:43.193921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe720d8-a234-42e2-94a9-dafa7e738fa2 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:43.202288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:43.204052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:43.204858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:43.205658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] PATH_INFO: `/v3/projects/e366a29135794c3c8d3a2a36d07c6bfb` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:43.221310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e366a29135794c3c8d3a2a36d07c6bfb)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:43.228747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:43.234842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:43.382677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e366a29135794c3c8d3a2a36d07c6bfb', 'request_id': 'req-a8fb31fd-8e2e-4127-8158-71707ddbc50b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:43.402270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e366a29135794c3c8d3a2a36d07c6bfb', 'request_id': 'req-a8fb31fd-8e2e-4127-8158-71707ddbc50b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:43.418976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e366a29135794c3c8d3a2a36d07c6bfb', 'request_id': 'req-a8fb31fd-8e2e-4127-8158-71707ddbc50b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:43.442599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] The token cache is being invalidate because project e366a29135794c3c8d3a2a36d07c6bfb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:47:43.443230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:43.444629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-a8fb31fd-8e2e-4127-8158-71707ddbc50b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:43.452279 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1183/2381] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:47:43 2022] DELETE /identity/v3/projects/e366a29135794c3c8d3a2a36d07c6bfb => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:43.463305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a46c3-e07e-42f9-a3cb-9970c010546b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:43.465744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a46c3-e07e-42f9-a3cb-9970c010546b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:43.465744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a46c3-e07e-42f9-a3cb-9970c010546b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:43.510826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-070a46c3-e07e-42f9-a3cb-9970c010546b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:43.515042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-070a46c3-e07e-42f9-a3cb-9970c010546b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:43.663845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1199/2382] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:43.676451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb8f74-e2e5-4e91-9a61-58beb426f76e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:43.676923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb8f74-e2e5-4e91-9a61-58beb426f76e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:43.677395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb8f74-e2e5-4e91-9a61-58beb426f76e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:43.711726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-39bb8f74-e2e5-4e91-9a61-58beb426f76e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:43.713326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-39bb8f74-e2e5-4e91-9a61-58beb426f76e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:43.738014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1184/2383] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:43.744856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a46c3-e07e-42f9-a3cb-9970c010546b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:43.779831 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a46c3-e07e-42f9-a3cb-9970c010546b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:43.802659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:43.804337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:43.805228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:43.806031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:43.807869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:47:43.808840 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:43.813604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:43.846833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1200/2384] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:47:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:47:43.857739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bb8f74-e2e5-4e91-9a61-58beb426f76e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:43.872575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bb8f74-e2e5-4e91-9a61-58beb426f76e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:43.881348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5028dc0a-b01a-4797-8f02-b02b3a51b2e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:43.882401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5028dc0a-b01a-4797-8f02-b02b3a51b2e3 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:43.882942 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5028dc0a-b01a-4797-8f02-b02b3a51b2e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:43.883434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5028dc0a-b01a-4797-8f02-b02b3a51b2e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:43.884597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5028dc0a-b01a-4797-8f02-b02b3a51b2e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:43.888481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5028dc0a-b01a-4797-8f02-b02b3a51b2e3 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:43.936744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1185/2385] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:43 2022] POST /identity/v3/projects => generated 363 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:47:43.949452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:43.970480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d9d930-7fbc-4ac6-85b8-3967311bf0a0 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.052779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0b48f9-7a5e-4e64-b95d-e689ec8aa2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.053402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b48f9-7a5e-4e64-b95d-e689ec8aa2a8 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.053683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b48f9-7a5e-4e64-b95d-e689ec8aa2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.053986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b48f9-7a5e-4e64-b95d-e689ec8aa2a8 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.054724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0b48f9-7a5e-4e64-b95d-e689ec8aa2a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:44.057658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0b48f9-7a5e-4e64-b95d-e689ec8aa2a8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:44.085318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1201/2386] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:43 2022] POST /identity/v3/users => generated 282 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:44.089731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5028dc0a-b01a-4797-8f02-b02b3a51b2e3 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:44.102559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5028dc0a-b01a-4797-8f02-b02b3a51b2e3 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.107981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f2b529-4283-407d-b4d6-f69a46673cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.108711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f2b529-4283-407d-b4d6-f69a46673cff admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.108888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f2b529-4283-407d-b4d6-f69a46673cff admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.109660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f2b529-4283-407d-b4d6-f69a46673cff admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.110405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f2b529-4283-407d-b4d6-f69a46673cff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:44.112242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f2b529-4283-407d-b4d6-f69a46673cff admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:44.125143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1186/2387] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:44 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:44.129563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0b48f9-7a5e-4e64-b95d-e689ec8aa2a8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:44.146422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0b48f9-7a5e-4e64-b95d-e689ec8aa2a8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.154305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485de377-eb64-4de5-8eff-81f2c337b1d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.155189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485de377-eb64-4de5-8eff-81f2c337b1d6 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.155476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485de377-eb64-4de5-8eff-81f2c337b1d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.155766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485de377-eb64-4de5-8eff-81f2c337b1d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.156639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485de377-eb64-4de5-8eff-81f2c337b1d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:44.159273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485de377-eb64-4de5-8eff-81f2c337b1d6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:44.177706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1202/2388] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:44 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:44.184470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f2b529-4283-407d-b4d6-f69a46673cff admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:44.204100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f2b529-4283-407d-b4d6-f69a46673cff admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.212200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42287663-a000-4110-9b1d-a0ad1273ed38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.213292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42287663-a000-4110-9b1d-a0ad1273ed38 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.213774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42287663-a000-4110-9b1d-a0ad1273ed38 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.214229 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42287663-a000-4110-9b1d-a0ad1273ed38 admin admin] PATH_INFO: `/v3/projects/78f2cc6934944fcb8c5065baef15d4be/users/940123f37193480a930236d2abbe03ac/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.243620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42287663-a000-4110-9b1d-a0ad1273ed38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78f2cc6934944fcb8c5065baef15d4be, user_id=940123f37193480a930236d2abbe03ac, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:44.248628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42287663-a000-4110-9b1d-a0ad1273ed38 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:44.272714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1187/2389] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:44 2022] PUT /identity/v3/projects/78f2cc6934944fcb8c5065baef15d4be/users/940123f37193480a930236d2abbe03ac/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:44.291640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8dbba-7546-49b0-876f-d0037557a0d2 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.292092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8dbba-7546-49b0-876f-d0037557a0d2 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.292412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8dbba-7546-49b0-876f-d0037557a0d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.323904 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:44.338513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2ee8dbba-7546-49b0-876f-d0037557a0d2 None None] MFA Rules not processed for user `940123f37193480a930236d2abbe03ac`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:44.340282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2ee8dbba-7546-49b0-876f-d0037557a0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:44.464457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1203/2390] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:44 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:44.469075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42287663-a000-4110-9b1d-a0ad1273ed38 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:44.545774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee8dbba-7546-49b0-876f-d0037557a0d2 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:44.548387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2ee8dbba-7546-49b0-876f-d0037557a0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:44.616876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42287663-a000-4110-9b1d-a0ad1273ed38 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.622685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb67030-f8ea-4b6d-b770-62b817c91c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.623340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb67030-f8ea-4b6d-b770-62b817c91c45 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.623814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb67030-f8ea-4b6d-b770-62b817c91c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.624241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb67030-f8ea-4b6d-b770-62b817c91c45 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.625223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb67030-f8ea-4b6d-b770-62b817c91c45 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:44.627485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb67030-f8ea-4b6d-b770-62b817c91c45 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:44.669967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1188/2391] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:44 2022] POST /identity/v3/projects => generated 365 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:44.671466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb67030-f8ea-4b6d-b770-62b817c91c45 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:44.678541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee8dbba-7546-49b0-876f-d0037557a0d2 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.692866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:44.769465 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.770966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.771669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.772518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.775763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:44.794928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:44.796372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb67030-f8ea-4b6d-b770-62b817c91c45 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.797117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:44.805690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e809513a-6720-4410-9044-bc7bf04df9aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.806940 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e809513a-6720-4410-9044-bc7bf04df9aa admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.807504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e809513a-6720-4410-9044-bc7bf04df9aa admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.808060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e809513a-6720-4410-9044-bc7bf04df9aa admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.809401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e809513a-6720-4410-9044-bc7bf04df9aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:44.811332 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1204/2392] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:44.813616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e809513a-6720-4410-9044-bc7bf04df9aa admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:44.875139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1189/2393] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:44 2022] POST /identity/v3/users => generated 289 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:44.877887 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:44.898556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc138c9f-68bc-43b1-ad99-6253e2f47c9e service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.907908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63245f87-e378-4c72-9c4b-d906fb39dd79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.908971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63245f87-e378-4c72-9c4b-d906fb39dd79 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.909472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63245f87-e378-4c72-9c4b-d906fb39dd79 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.909974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63245f87-e378-4c72-9c4b-d906fb39dd79 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:44.911478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63245f87-e378-4c72-9c4b-d906fb39dd79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:44.914667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63245f87-e378-4c72-9c4b-d906fb39dd79 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:44.941811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1205/2394] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:44 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:44.944770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e809513a-6720-4410-9044-bc7bf04df9aa admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:44.960888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e809513a-6720-4410-9044-bc7bf04df9aa admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:44.969647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b98f2e-a96e-4dc4-befe-0931f220915a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:44.971108 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b98f2e-a96e-4dc4-befe-0931f220915a admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:44.971733 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b98f2e-a96e-4dc4-befe-0931f220915a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:44.972330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b98f2e-a96e-4dc4-befe-0931f220915a admin admin] PATH_INFO: `/v3/projects/bdf903c0cdeb4807903097c85f4f205a/users/327eda785ebf46719078d3a2ac678552/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:45.002652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b98f2e-a96e-4dc4-befe-0931f220915a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdf903c0cdeb4807903097c85f4f205a, user_id=327eda785ebf46719078d3a2ac678552, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:45.007238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b98f2e-a96e-4dc4-befe-0931f220915a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:45.037114 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1190/2395] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:44 2022] PUT /identity/v3/projects/bdf903c0cdeb4807903097c85f4f205a/users/327eda785ebf46719078d3a2ac678552/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:45.042133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63245f87-e378-4c72-9c4b-d906fb39dd79 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:45.058335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63245f87-e378-4c72-9c4b-d906fb39dd79 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:45.182442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:45.205036 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000f4c8f-f23f-4d0a-86e6-ed07d58268a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:45.206433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000f4c8f-f23f-4d0a-86e6-ed07d58268a4 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:45.206612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000f4c8f-f23f-4d0a-86e6-ed07d58268a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:45.206995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000f4c8f-f23f-4d0a-86e6-ed07d58268a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:45.207908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000f4c8f-f23f-4d0a-86e6-ed07d58268a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:45.210939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000f4c8f-f23f-4d0a-86e6-ed07d58268a4 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:45.231811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1206/2396] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:45 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:45.238296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b98f2e-a96e-4dc4-befe-0931f220915a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:45.258136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b98f2e-a96e-4dc4-befe-0931f220915a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:45.277421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:45.278483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:45.278969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:45.279614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] PATH_INFO: `/v3/projects/bdf903c0cdeb4807903097c85f4f205a/users/327eda785ebf46719078d3a2ac678552/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:45.284271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdf903c0cdeb4807903097c85f4f205a, user_id=327eda785ebf46719078d3a2ac678552, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:45.289845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:45.321430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1191/2397] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:45 2022] PUT /identity/v3/projects/bdf903c0cdeb4807903097c85f4f205a/users/327eda785ebf46719078d3a2ac678552/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:45.332689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe97e11d-3838-406d-be76-9a05f1af36a3 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:45.333432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe97e11d-3838-406d-be76-9a05f1af36a3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:45.333728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe97e11d-3838-406d-be76-9a05f1af36a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:45.396827 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-fe97e11d-3838-406d-be76-9a05f1af36a3 None None] MFA Rules not processed for user `327eda785ebf46719078d3a2ac678552`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:45.399449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-fe97e11d-3838-406d-be76-9a05f1af36a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:45.563628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1207/2398] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:45 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:45.575230 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:45.580231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:45.715983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c319cf2-e613-45df-bef9-8d79b213f124 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:45.731069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c239dc-2d8f-447f-a1cd-8592540a921f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:45.732746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c239dc-2d8f-447f-a1cd-8592540a921f service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:45.733451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c239dc-2d8f-447f-a1cd-8592540a921f service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:45.735478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c239dc-2d8f-447f-a1cd-8592540a921f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:45.759847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c239dc-2d8f-447f-a1cd-8592540a921f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:45.761585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c239dc-2d8f-447f-a1cd-8592540a921f service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:45.790857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1192/2399] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:45 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:45.938711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe97e11d-3838-406d-be76-9a05f1af36a3 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:45.956926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe97e11d-3838-406d-be76-9a05f1af36a3 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:45.963954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded52d6-d651-45bf-8be2-ce89586cbe5f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:45.965049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded52d6-d651-45bf-8be2-ce89586cbe5f service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:45.966678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded52d6-d651-45bf-8be2-ce89586cbe5f service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:45.967160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded52d6-d651-45bf-8be2-ce89586cbe5f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:46.076084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fded52d6-d651-45bf-8be2-ce89586cbe5f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:46.078702 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fded52d6-d651-45bf-8be2-ce89586cbe5f service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:46.095699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1208/2400] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:45 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:48.952200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749f933-0315-4610-a4d5-20ff5f08a93a None None] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:48.952842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749f933-0315-4610-a4d5-20ff5f08a93a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:48.953658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749f933-0315-4610-a4d5-20ff5f08a93a None None] PATH_INFO: `/` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:48.956440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1193/2401] 10.222.0.77 () {54 vars in 921 bytes} [Mon Aug 29 22:47:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 29 22:47:49.000175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7558759-053e-4d98-b212-b45b363a50f7 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:49.001118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7558759-053e-4d98-b212-b45b363a50f7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:49.001931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7558759-053e-4d98-b212-b45b363a50f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:49.034104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d7558759-053e-4d98-b212-b45b363a50f7 None None] MFA Rules not processed for user `b6f80303c62847ca9a506d58d9b6f396`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:49.036223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d7558759-053e-4d98-b212-b45b363a50f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:49.104053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e749f933-0315-4610-a4d5-20ff5f08a93a None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:49.106346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e749f933-0315-4610-a4d5-20ff5f08a93a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:49.209429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1209/2402] 10.222.0.77 () {58 vars in 1018 bytes} [Mon Aug 29 22:47:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:49.226799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7558759-053e-4d98-b212-b45b363a50f7 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:49.228999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d7558759-053e-4d98-b212-b45b363a50f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:49.245144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:49.251350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7558759-053e-4d98-b212-b45b363a50f7 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:49.278918 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66371c87-4916-4d62-b48c-974757f94554 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:49.280078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66371c87-4916-4d62-b48c-974757f94554 service placement] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:49.280533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66371c87-4916-4d62-b48c-974757f94554 service placement] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:49.281824 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66371c87-4916-4d62-b48c-974757f94554 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:49.287303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e749f933-0315-4610-a4d5-20ff5f08a93a None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:49.298103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:49.300125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:49.301188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:49.301823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] PATH_INFO: `/v3/users/8b46b3fa1df24c80ab7315c4ff44c064` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:49.304829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b46b3fa1df24c80ab7315c4ff44c064)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:49.309550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:49.316753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66371c87-4916-4d62-b48c-974757f94554 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:49.333047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66371c87-4916-4d62-b48c-974757f94554 service placement] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:49.340362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:49.367426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1210/2403] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:49.450365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b46b3fa1df24c80ab7315c4ff44c064'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.451199 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b46b3fa1df24c80ab7315c4ff44c064'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.469749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b46b3fa1df24c80ab7315c4ff44c064'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.491037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b46b3fa1df24c80ab7315c4ff44c064'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.526752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-9537e2dd-f2c6-4d41-a001-62b587c37ac2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b46b3fa1df24c80ab7315c4ff44c064'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.536902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66371c87-4916-4d62-b48c-974757f94554 service placement] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:49.539688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1194/2404] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:49 2022] DELETE /identity/v3/users/8b46b3fa1df24c80ab7315c4ff44c064 => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:49.556933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66371c87-4916-4d62-b48c-974757f94554 service placement] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:49.675907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:49.677316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:49.677730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:49.678230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] PATH_INFO: `/v3/users/940123f37193480a930236d2abbe03ac` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:49.680600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] RBAC: Authorizing `identity:delete_user(user_id=940123f37193480a930236d2abbe03ac)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:49.683947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:49.782472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '940123f37193480a930236d2abbe03ac'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.797394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '940123f37193480a930236d2abbe03ac'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.798561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '940123f37193480a930236d2abbe03ac'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.799860 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '940123f37193480a930236d2abbe03ac'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.813106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '940123f37193480a930236d2abbe03ac'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:49.848226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1211/2405] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:49 2022] DELETE /identity/v3/users/940123f37193480a930236d2abbe03ac => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:49.854985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af18a11-2f8a-497e-9ca0-39da6a6d0c1b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:49.856016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af18a11-2f8a-497e-9ca0-39da6a6d0c1b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:49.856016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af18a11-2f8a-497e-9ca0-39da6a6d0c1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:49.892952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:49.893414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9af18a11-2f8a-497e-9ca0-39da6a6d0c1b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:49.894643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:49.895720 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9af18a11-2f8a-497e-9ca0-39da6a6d0c1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:50.023571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48132d7-c02a-4f86-9cc0-ff6fcdfbd70f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:50.029131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:50.029986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.030265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.030540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] PATH_INFO: `/v3/projects/8ad722485ac34d1796f03003fbcde07a` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.032784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ad722485ac34d1796f03003fbcde07a)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:50.035063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:50.042596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1195/2406] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:50.054762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af18a11-2f8a-497e-9ca0-39da6a6d0c1b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:50.056241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9af18a11-2f8a-497e-9ca0-39da6a6d0c1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:50.058982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:50.167446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ad722485ac34d1796f03003fbcde07a', 'request_id': 'req-f5b7b993-5a58-4c36-8dfd-264562882037'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:50.181187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ad722485ac34d1796f03003fbcde07a', 'request_id': 'req-f5b7b993-5a58-4c36-8dfd-264562882037'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:50.191915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ad722485ac34d1796f03003fbcde07a', 'request_id': 'req-f5b7b993-5a58-4c36-8dfd-264562882037'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:50.192562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af18a11-2f8a-497e-9ca0-39da6a6d0c1b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:50.223154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] The token cache is being invalidate because project 8ad722485ac34d1796f03003fbcde07a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:47:50.224149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:50.225422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f5b7b993-5a58-4c36-8dfd-264562882037 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:50.230834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1212/2407] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:47:49 2022] DELETE /identity/v3/projects/8ad722485ac34d1796f03003fbcde07a => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:50.236201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49faf7af-4bad-4b82-963f-dca310c0149f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.236661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49faf7af-4bad-4b82-963f-dca310c0149f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.237138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49faf7af-4bad-4b82-963f-dca310c0149f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.269148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-49faf7af-4bad-4b82-963f-dca310c0149f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:50.271836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-49faf7af-4bad-4b82-963f-dca310c0149f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:50.284927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:50.295998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:50.297213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.297578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.298433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.302907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:50.378804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1213/2408] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:50.385021 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c7e1f3-2f9e-4a44-b64a-884d8c2c94db None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.386461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c7e1f3-2f9e-4a44-b64a-884d8c2c94db None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.387295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c7e1f3-2f9e-4a44-b64a-884d8c2c94db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.419387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:50.421343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:50.425258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-67c7e1f3-2f9e-4a44-b64a-884d8c2c94db None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:50.427051 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-67c7e1f3-2f9e-4a44-b64a-884d8c2c94db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:50.432778 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1196/2409] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:50.440127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1214/2410] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:50.448073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:50.462692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc58d4a0-a1f8-40ef-92d7-bfab096610bd service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:50.471135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:50.472056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.472449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.472847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.473787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:47:50.474255 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:50.476492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:50.509803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1197/2411] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:47:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:47:50.516792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c7e1f3-2f9e-4a44-b64a-884d8c2c94db None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:50.536517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c7e1f3-2f9e-4a44-b64a-884d8c2c94db None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:50.543027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b39572-5e8c-42f4-a573-6fe3b248537b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:50.544304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b39572-5e8c-42f4-a573-6fe3b248537b admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.544780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b39572-5e8c-42f4-a573-6fe3b248537b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.545614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b39572-5e8c-42f4-a573-6fe3b248537b admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.547711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b39572-5e8c-42f4-a573-6fe3b248537b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:50.551609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b39572-5e8c-42f4-a573-6fe3b248537b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:50.623023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1215/2412] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:50 2022] POST /identity/v3/projects => generated 371 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:50.624510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:50.645762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68409e7c-3c41-476c-b806-1466e1e04538 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:50.769575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f10b3-b239-488e-aee5-58c82d016ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:50.770755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f10b3-b239-488e-aee5-58c82d016ffc admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.771955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f10b3-b239-488e-aee5-58c82d016ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.772441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f10b3-b239-488e-aee5-58c82d016ffc admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.775682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f10b3-b239-488e-aee5-58c82d016ffc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:50.781525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f10b3-b239-488e-aee5-58c82d016ffc admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:50.847981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1198/2413] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:50 2022] POST /identity/v3/users => generated 286 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:50.854461 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b39572-5e8c-42f4-a573-6fe3b248537b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:50.890927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b39572-5e8c-42f4-a573-6fe3b248537b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:50.900077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6c6b30-95f6-4862-953b-91f9508d703b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:50.902446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c6b30-95f6-4862-953b-91f9508d703b admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.903274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c6b30-95f6-4862-953b-91f9508d703b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.904110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c6b30-95f6-4862-953b-91f9508d703b admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.905712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6c6b30-95f6-4862-953b-91f9508d703b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:50.910026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6c6b30-95f6-4862-953b-91f9508d703b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:50.926792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1216/2414] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:50 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:50.933064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f10b3-b239-488e-aee5-58c82d016ffc admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:50.947189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f10b3-b239-488e-aee5-58c82d016ffc admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:50.954626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fefcd3-36fe-4f3d-97d9-033a396b541d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:50.955375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fefcd3-36fe-4f3d-97d9-033a396b541d admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:50.955754 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fefcd3-36fe-4f3d-97d9-033a396b541d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:50.956252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fefcd3-36fe-4f3d-97d9-033a396b541d admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:50.957754 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fefcd3-36fe-4f3d-97d9-033a396b541d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:50.959976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fefcd3-36fe-4f3d-97d9-033a396b541d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:50.991128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1199/2415] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:50 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:51.001014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6c6b30-95f6-4862-953b-91f9508d703b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:51.018210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6c6b30-95f6-4862-953b-91f9508d703b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:51.026867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:51.027937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:51.028377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:51.028863 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] PATH_INFO: `/v3/projects/346ed3941744457aa077b7b9fe51bf54/users/77e911424c1b4e9cb7720370822b46d3/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:51.054799 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=346ed3941744457aa077b7b9fe51bf54, user_id=77e911424c1b4e9cb7720370822b46d3, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:51.059921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:51.082550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1217/2416] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:50 2022] PUT /identity/v3/projects/346ed3941744457aa077b7b9fe51bf54/users/77e911424c1b4e9cb7720370822b46d3/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:51.096672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dba059-b7c4-4945-a86b-80100174b6e4 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:51.097457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dba059-b7c4-4945-a86b-80100174b6e4 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:51.097457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dba059-b7c4-4945-a86b-80100174b6e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:51.130579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c2dba059-b7c4-4945-a86b-80100174b6e4 None None] MFA Rules not processed for user `77e911424c1b4e9cb7720370822b46d3`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:51.132026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c2dba059-b7c4-4945-a86b-80100174b6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:51.222718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:51.230178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:51.255295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:51.323296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1200/2417] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:51 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:51.337519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dba059-b7c4-4945-a86b-80100174b6e4 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:51.341656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c2dba059-b7c4-4945-a86b-80100174b6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:51.349040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:51.410061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf34ced-9b4d-41b4-8730-cd75760e3d9e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:51.417406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:51.418500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:51.419028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:51.419501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] PATH_INFO: `/v3/projects/78f2cc6934944fcb8c5065baef15d4be` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:51.428540 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] RBAC: Authorizing `identity:delete_project(project_id=78f2cc6934944fcb8c5065baef15d4be)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:51.433634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:51.509963 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dba059-b7c4-4945-a86b-80100174b6e4 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:51.536760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78f2cc6934944fcb8c5065baef15d4be', 'request_id': 'req-e4b61a88-c210-4bcb-9b00-c7937b8de13b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.553499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78f2cc6934944fcb8c5065baef15d4be', 'request_id': 'req-e4b61a88-c210-4bcb-9b00-c7937b8de13b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.566598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78f2cc6934944fcb8c5065baef15d4be', 'request_id': 'req-e4b61a88-c210-4bcb-9b00-c7937b8de13b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.594907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] The token cache is being invalidate because project 78f2cc6934944fcb8c5065baef15d4be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:47:51.595307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.596913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.600694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1218/2418] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:47:51 2022] DELETE /identity/v3/projects/78f2cc6934944fcb8c5065baef15d4be => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:51.606338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:51.608838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:51.676070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:51.677252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:51.677739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:51.678209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:51.682216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:51.753965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b61a88-c210-4bcb-9b00-c7937b8de13b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:51.762922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:51.764105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:51.764604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:51.765098 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] PATH_INFO: `/v3/users/327eda785ebf46719078d3a2ac678552` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:51.767694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=327eda785ebf46719078d3a2ac678552)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:51.771601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:51.811488 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:51.814111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:51.847895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1201/2419] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:51 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 512 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:51.856929 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '327eda785ebf46719078d3a2ac678552'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.889374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '327eda785ebf46719078d3a2ac678552'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.890896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '327eda785ebf46719078d3a2ac678552'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.891983 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '327eda785ebf46719078d3a2ac678552'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.908663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '327eda785ebf46719078d3a2ac678552'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:51.923912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:51.959402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1219/2420] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:51 2022] DELETE /identity/v3/users/327eda785ebf46719078d3a2ac678552 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:52.013205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:52.014725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:52.040916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe9f35d-9669-401b-95c5-5a7a151f1960 service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:52.080957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:52.148173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:52.149383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:52.149932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:52.150372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] PATH_INFO: `/v3/users/70da750fbe764462b50dc0fcef3eb0e6` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:52.152202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] RBAC: Authorizing `identity:delete_user(user_id=70da750fbe764462b50dc0fcef3eb0e6)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:52.154933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:52.258418 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70da750fbe764462b50dc0fcef3eb0e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.259929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70da750fbe764462b50dc0fcef3eb0e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.276071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70da750fbe764462b50dc0fcef3eb0e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.294854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70da750fbe764462b50dc0fcef3eb0e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.327152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70da750fbe764462b50dc0fcef3eb0e6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.335168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1202/2421] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:52 2022] DELETE /identity/v3/users/70da750fbe764462b50dc0fcef3eb0e6 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:52.481109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:52.490610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:52.565186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2143197d-d6cf-4823-8b9f-23b0780ad9fd admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:52.679327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:52.680136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:52.680548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:52.680979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] PATH_INFO: `/v3/projects/bdf903c0cdeb4807903097c85f4f205a` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:52.682584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdf903c0cdeb4807903097c85f4f205a)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:52.685543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:52.700450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:52.703989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:52.729514 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6fbb18-2adc-4935-8c77-d5c015b74875 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:52.739294 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:52.741325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:52.741325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:52.741790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] PATH_INFO: `/v3/projects/cf0dd257dea04aca845b17f51e1c200e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:52.745022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf0dd257dea04aca845b17f51e1c200e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:52.748047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:52.783384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdf903c0cdeb4807903097c85f4f205a', 'request_id': 'req-edae282f-b346-4024-a601-87b70480e9a2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.803420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdf903c0cdeb4807903097c85f4f205a', 'request_id': 'req-edae282f-b346-4024-a601-87b70480e9a2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.829629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdf903c0cdeb4807903097c85f4f205a', 'request_id': 'req-edae282f-b346-4024-a601-87b70480e9a2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.851497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] The token cache is being invalidate because project bdf903c0cdeb4807903097c85f4f205a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:47:52.853627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.855875 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edae282f-b346-4024-a601-87b70480e9a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.857704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf0dd257dea04aca845b17f51e1c200e', 'request_id': 'req-8a341cee-8b7b-4735-8942-e76e01b6801e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.862903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1220/2422] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:47:52 2022] DELETE /identity/v3/projects/bdf903c0cdeb4807903097c85f4f205a => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:52.869779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecd2bef-06fa-419e-b470-0b87ddd132a7 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:52.870049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf0dd257dea04aca845b17f51e1c200e', 'request_id': 'req-8a341cee-8b7b-4735-8942-e76e01b6801e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.870261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecd2bef-06fa-419e-b470-0b87ddd132a7 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:52.870522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecd2bef-06fa-419e-b470-0b87ddd132a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:52.882455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf0dd257dea04aca845b17f51e1c200e', 'request_id': 'req-8a341cee-8b7b-4735-8942-e76e01b6801e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.904645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] The token cache is being invalidate because project cf0dd257dea04aca845b17f51e1c200e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:47:52.905179 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.906309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a341cee-8b7b-4735-8942-e76e01b6801e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:52.908864 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1203/2423] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:47:52 2022] DELETE /identity/v3/projects/cf0dd257dea04aca845b17f51e1c200e => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:52.909837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1ecd2bef-06fa-419e-b470-0b87ddd132a7 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:52.912240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1ecd2bef-06fa-419e-b470-0b87ddd132a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:52.916403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4db0fb-0e7f-4b8e-8d11-abfb1fc827a5 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:52.916403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4db0fb-0e7f-4b8e-8d11-abfb1fc827a5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:52.917481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4db0fb-0e7f-4b8e-8d11-abfb1fc827a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:52.960451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-8e4db0fb-0e7f-4b8e-8d11-abfb1fc827a5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:52.962410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8e4db0fb-0e7f-4b8e-8d11-abfb1fc827a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:53.001220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:53.043310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1221/2424] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:53.057038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2b79-ef25-470b-85c5-781cf4ba2d43 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.057530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2b79-ef25-470b-85c5-781cf4ba2d43 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.057936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2b79-ef25-470b-85c5-781cf4ba2d43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.061505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1204/2425] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:53.070924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e846b9c-8ad3-43ba-83ab-3931a470764f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.071296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e846b9c-8ad3-43ba-83ab-3931a470764f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.071603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e846b9c-8ad3-43ba-83ab-3931a470764f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.084479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-557d2b79-ef25-470b-85c5-781cf4ba2d43 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:53.086124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-557d2b79-ef25-470b-85c5-781cf4ba2d43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:53.097057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1222/2426] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:53.103006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d2b79-ef25-470b-85c5-781cf4ba2d43 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.107941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-8e846b9c-8ad3-43ba-83ab-3931a470764f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:53.110633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8e846b9c-8ad3-43ba-83ab-3931a470764f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:53.117544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d2b79-ef25-470b-85c5-781cf4ba2d43 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.126230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.127311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.127579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1205/2427] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:53.127818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.128053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.128788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:47:53.129184 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.131046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.131927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e846b9c-8ad3-43ba-83ab-3931a470764f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.142163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e846b9c-8ad3-43ba-83ab-3931a470764f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.143856 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:53.149535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.150693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.151153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.151554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.152335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:47:53.152741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.154679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.164241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1223/2428] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:47:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:47:53.166593 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.181966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1206/2429] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:47:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:47:53.183804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.184753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a460fa7a-bc6a-4554-b53e-c7ab0e8950ce admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.192405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592e4683-d92c-4ebc-89d0-1d9ff774a855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.193245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e4683-d92c-4ebc-89d0-1d9ff774a855 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.193525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e4683-d92c-4ebc-89d0-1d9ff774a855 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.193908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e4683-d92c-4ebc-89d0-1d9ff774a855 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.195023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592e4683-d92c-4ebc-89d0-1d9ff774a855 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.197994 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592e4683-d92c-4ebc-89d0-1d9ff774a855 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.200253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcb3824-a17d-4297-9ce0-f77e20c7cb4f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.206382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e155316-f196-41cb-a301-fb7563348805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.207077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e155316-f196-41cb-a301-fb7563348805 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.207346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e155316-f196-41cb-a301-fb7563348805 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.207633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e155316-f196-41cb-a301-fb7563348805 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.208261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e155316-f196-41cb-a301-fb7563348805 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.210394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e155316-f196-41cb-a301-fb7563348805 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.236694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1224/2430] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/projects => generated 367 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:53.243217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592e4683-d92c-4ebc-89d0-1d9ff774a855 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.261954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1207/2431] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/projects => generated 381 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:47:53.273734 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e155316-f196-41cb-a301-fb7563348805 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.350311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592e4683-d92c-4ebc-89d0-1d9ff774a855 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.363535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e155316-f196-41cb-a301-fb7563348805 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.369594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c6aa49-80c8-45ff-9f0b-d2ad09b010a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.370535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6aa49-80c8-45ff-9f0b-d2ad09b010a6 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.371060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6aa49-80c8-45ff-9f0b-d2ad09b010a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.371403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6aa49-80c8-45ff-9f0b-d2ad09b010a6 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.372069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c6aa49-80c8-45ff-9f0b-d2ad09b010a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.376098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c6aa49-80c8-45ff-9f0b-d2ad09b010a6 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.411518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1208/2432] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/users => generated 291 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:53.417001 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c6aa49-80c8-45ff-9f0b-d2ad09b010a6 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.423673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f21a69f-d94d-466d-90a5-ac9228e8acff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.424303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21a69f-d94d-466d-90a5-ac9228e8acff admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.424589 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21a69f-d94d-466d-90a5-ac9228e8acff admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.424879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21a69f-d94d-466d-90a5-ac9228e8acff admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.425552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f21a69f-d94d-466d-90a5-ac9228e8acff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.428213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f21a69f-d94d-466d-90a5-ac9228e8acff admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.431206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c6aa49-80c8-45ff-9f0b-d2ad09b010a6 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.438855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33966bd0-dee3-4db7-8279-40c0defc6d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.439683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33966bd0-dee3-4db7-8279-40c0defc6d43 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.439971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33966bd0-dee3-4db7-8279-40c0defc6d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.440366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33966bd0-dee3-4db7-8279-40c0defc6d43 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.441037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33966bd0-dee3-4db7-8279-40c0defc6d43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.443208 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33966bd0-dee3-4db7-8279-40c0defc6d43 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.459990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1225/2433] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/users => generated 284 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:47:53.463284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1209/2434] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:53.465066 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f21a69f-d94d-466d-90a5-ac9228e8acff admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.467028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33966bd0-dee3-4db7-8279-40c0defc6d43 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.480464 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33966bd0-dee3-4db7-8279-40c0defc6d43 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.481137 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f21a69f-d94d-466d-90a5-ac9228e8acff admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.487785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc98778-2347-44a4-a568-e0d5a917f086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.488020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ed4212-7ec2-4a10-95f6-01eab3662980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.488422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc98778-2347-44a4-a568-e0d5a917f086 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.488707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc98778-2347-44a4-a568-e0d5a917f086 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.488900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed4212-7ec2-4a10-95f6-01eab3662980 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.489090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc98778-2347-44a4-a568-e0d5a917f086 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.489287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed4212-7ec2-4a10-95f6-01eab3662980 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.489628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed4212-7ec2-4a10-95f6-01eab3662980 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.489985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc98778-2347-44a4-a568-e0d5a917f086 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.490287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ed4212-7ec2-4a10-95f6-01eab3662980 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.491707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc98778-2347-44a4-a568-e0d5a917f086 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.491924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ed4212-7ec2-4a10-95f6-01eab3662980 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.504874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1210/2435] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:53.506518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1226/2436] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:53.509870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ed4212-7ec2-4a10-95f6-01eab3662980 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.512451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc98778-2347-44a4-a568-e0d5a917f086 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.530382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc98778-2347-44a4-a568-e0d5a917f086 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.537035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc82f9bd-96d2-4a5f-b9f8-d2ff956dcc4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.537660 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc82f9bd-96d2-4a5f-b9f8-d2ff956dcc4e admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.537806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ed4212-7ec2-4a10-95f6-01eab3662980 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.538056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc82f9bd-96d2-4a5f-b9f8-d2ff956dcc4e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.538297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc82f9bd-96d2-4a5f-b9f8-d2ff956dcc4e admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.539292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc82f9bd-96d2-4a5f-b9f8-d2ff956dcc4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.541202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc82f9bd-96d2-4a5f-b9f8-d2ff956dcc4e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.547362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd67c658-151d-49ed-b2f0-c1b01de67498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.548499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd67c658-151d-49ed-b2f0-c1b01de67498 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.548986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd67c658-151d-49ed-b2f0-c1b01de67498 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.549467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd67c658-151d-49ed-b2f0-c1b01de67498 admin admin] PATH_INFO: `/v3/projects/c5ccca60312e404e97077965d5ec0ee5/users/1aee5fe10f074ec18305b843233f475d/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.559941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1227/2437] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:53.564804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc82f9bd-96d2-4a5f-b9f8-d2ff956dcc4e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.568854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd67c658-151d-49ed-b2f0-c1b01de67498 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5ccca60312e404e97077965d5ec0ee5, user_id=1aee5fe10f074ec18305b843233f475d, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.571604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd67c658-151d-49ed-b2f0-c1b01de67498 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.578707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc82f9bd-96d2-4a5f-b9f8-d2ff956dcc4e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.586295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c5f197-df94-4f3a-ba77-4a57f5b08ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.587632 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5f197-df94-4f3a-ba77-4a57f5b08ce6 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.588132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5f197-df94-4f3a-ba77-4a57f5b08ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.589083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5f197-df94-4f3a-ba77-4a57f5b08ce6 admin admin] PATH_INFO: `/v3/projects/3e4072234bbc4a03b793aa756930330b/users/5e83d357404b47c28e4c777998cbfdd2/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.592797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1211/2438] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:53 2022] PUT /identity/v3/projects/c5ccca60312e404e97077965d5ec0ee5/users/1aee5fe10f074ec18305b843233f475d/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:53.609619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720748b-13b5-4a45-817e-9579046b22df None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.610035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720748b-13b5-4a45-817e-9579046b22df None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.610492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720748b-13b5-4a45-817e-9579046b22df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.611858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c5f197-df94-4f3a-ba77-4a57f5b08ce6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e4072234bbc4a03b793aa756930330b, user_id=5e83d357404b47c28e4c777998cbfdd2, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.614811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c5f197-df94-4f3a-ba77-4a57f5b08ce6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.635585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1228/2439] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:53 2022] PUT /identity/v3/projects/3e4072234bbc4a03b793aa756930330b/users/5e83d357404b47c28e4c777998cbfdd2/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:53.651942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72eaae1-08d2-441e-9cfb-f1dea2efb94f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.652388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72eaae1-08d2-441e-9cfb-f1dea2efb94f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.652822 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72eaae1-08d2-441e-9cfb-f1dea2efb94f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.658971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-6720748b-13b5-4a45-817e-9579046b22df None None] MFA Rules not processed for user `1aee5fe10f074ec18305b843233f475d`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:53.660618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6720748b-13b5-4a45-817e-9579046b22df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:53.695546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:53.705616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e72eaae1-08d2-441e-9cfb-f1dea2efb94f None None] MFA Rules not processed for user `5e83d357404b47c28e4c777998cbfdd2`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:53.707466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e72eaae1-08d2-441e-9cfb-f1dea2efb94f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:53.730240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:53.807350 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1212/2440] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:53.813736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6720748b-13b5-4a45-817e-9579046b22df None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.824857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1229/2441] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:53.830477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72eaae1-08d2-441e-9cfb-f1dea2efb94f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:53.935992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72eaae1-08d2-441e-9cfb-f1dea2efb94f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.943136 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6720748b-13b5-4a45-817e-9579046b22df None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:53.943820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd97015-bb0e-443e-bdb0-7e5247bf5dd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.944730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd97015-bb0e-443e-bdb0-7e5247bf5dd3 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.945108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd97015-bb0e-443e-bdb0-7e5247bf5dd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.945390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd97015-bb0e-443e-bdb0-7e5247bf5dd3 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.946097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd97015-bb0e-443e-bdb0-7e5247bf5dd3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.948106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd97015-bb0e-443e-bdb0-7e5247bf5dd3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:53.951954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ac236d-82e6-4303-90a9-42861b99c8e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:53.955185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac236d-82e6-4303-90a9-42861b99c8e9 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:53.955472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac236d-82e6-4303-90a9-42861b99c8e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:53.955746 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac236d-82e6-4303-90a9-42861b99c8e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:53.956457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ac236d-82e6-4303-90a9-42861b99c8e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:53.958535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ac236d-82e6-4303-90a9-42861b99c8e9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.003179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1230/2442] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/projects => generated 369 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:47:54.010832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1213/2443] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:53 2022] POST /identity/v3/projects => generated 381 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:47:54.011049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd97015-bb0e-443e-bdb0-7e5247bf5dd3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.012363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ac236d-82e6-4303-90a9-42861b99c8e9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.120128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ac236d-82e6-4303-90a9-42861b99c8e9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.126763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a488e3e-5c1f-4212-aacb-d78cc0ebe28d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.127479 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a488e3e-5c1f-4212-aacb-d78cc0ebe28d admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.127786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a488e3e-5c1f-4212-aacb-d78cc0ebe28d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.128134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a488e3e-5c1f-4212-aacb-d78cc0ebe28d admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.128787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a488e3e-5c1f-4212-aacb-d78cc0ebe28d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.130497 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd97015-bb0e-443e-bdb0-7e5247bf5dd3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.131284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a488e3e-5c1f-4212-aacb-d78cc0ebe28d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.139627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f53947f-ed2b-45c9-8565-3fe2459df386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.140738 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53947f-ed2b-45c9-8565-3fe2459df386 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.141974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53947f-ed2b-45c9-8565-3fe2459df386 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.141974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53947f-ed2b-45c9-8565-3fe2459df386 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.143136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f53947f-ed2b-45c9-8565-3fe2459df386 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.147646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f53947f-ed2b-45c9-8565-3fe2459df386 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.157233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1214/2444] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:54 2022] POST /identity/v3/users => generated 297 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:54.161753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a488e3e-5c1f-4212-aacb-d78cc0ebe28d admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.173790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a488e3e-5c1f-4212-aacb-d78cc0ebe28d admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.180225 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d58588b-a085-4588-99b2-c8bfb6ff1883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.180959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d58588b-a085-4588-99b2-c8bfb6ff1883 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.181972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d58588b-a085-4588-99b2-c8bfb6ff1883 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.181972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d58588b-a085-4588-99b2-c8bfb6ff1883 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.182391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d58588b-a085-4588-99b2-c8bfb6ff1883 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.184152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d58588b-a085-4588-99b2-c8bfb6ff1883 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.191252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1231/2445] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:54 2022] POST /identity/v3/users => generated 291 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:47:54.194447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f53947f-ed2b-45c9-8565-3fe2459df386 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.199834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1215/2446] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:54 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:54.204772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d58588b-a085-4588-99b2-c8bfb6ff1883 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.216348 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f53947f-ed2b-45c9-8565-3fe2459df386 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.223979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3702ebf1-2356-45f3-966e-676b705869d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.224762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702ebf1-2356-45f3-966e-676b705869d1 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.225122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702ebf1-2356-45f3-966e-676b705869d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.225481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702ebf1-2356-45f3-966e-676b705869d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.226208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3702ebf1-2356-45f3-966e-676b705869d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.228491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3702ebf1-2356-45f3-966e-676b705869d1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.234593 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d58588b-a085-4588-99b2-c8bfb6ff1883 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.239771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e477fd-92d3-4f39-ad90-12b2d49607e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.240446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e477fd-92d3-4f39-ad90-12b2d49607e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.240741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e477fd-92d3-4f39-ad90-12b2d49607e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.241004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e477fd-92d3-4f39-ad90-12b2d49607e1 admin admin] PATH_INFO: `/v3/projects/2047dcfb8e0240e2aee9991fd78904ce/users/6e5fa522caa04c92801a2295cfc7f729/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.243340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1232/2447] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:54 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:54.247818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3702ebf1-2356-45f3-966e-676b705869d1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.261978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3702ebf1-2356-45f3-966e-676b705869d1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.268026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e477fd-92d3-4f39-ad90-12b2d49607e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2047dcfb8e0240e2aee9991fd78904ce, user_id=6e5fa522caa04c92801a2295cfc7f729, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.269151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6b9c0c-394e-46d3-b062-ae138ba2bfcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.269888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b9c0c-394e-46d3-b062-ae138ba2bfcf admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.270288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b9c0c-394e-46d3-b062-ae138ba2bfcf admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.270642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b9c0c-394e-46d3-b062-ae138ba2bfcf admin admin] PATH_INFO: `/v3/projects/6d26233fa2c14a5d9964e02c43b2e403/users/0e683da52fc94eddad32413aab85c56d/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.272029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e477fd-92d3-4f39-ad90-12b2d49607e1 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.285029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:54.291510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6b9c0c-394e-46d3-b062-ae138ba2bfcf admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d26233fa2c14a5d9964e02c43b2e403, user_id=0e683da52fc94eddad32413aab85c56d, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.295072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1216/2448] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:54 2022] PUT /identity/v3/projects/2047dcfb8e0240e2aee9991fd78904ce/users/6e5fa522caa04c92801a2295cfc7f729/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:54.296846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6b9c0c-394e-46d3-b062-ae138ba2bfcf admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.300444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e477fd-92d3-4f39-ad90-12b2d49607e1 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.302334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:54.320097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1233/2449] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:54 2022] PUT /identity/v3/projects/6d26233fa2c14a5d9964e02c43b2e403/users/0e683da52fc94eddad32413aab85c56d/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:54.325147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6b9c0c-394e-46d3-b062-ae138ba2bfcf admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.391973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e477fd-92d3-4f39-ad90-12b2d49607e1 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.419450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6b9c0c-394e-46d3-b062-ae138ba2bfcf admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.427907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4515b1cc-72a3-4ba0-bcba-6e939421531d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.428951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4515b1cc-72a3-4ba0-bcba-6e939421531d admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.429432 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4515b1cc-72a3-4ba0-bcba-6e939421531d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.429900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4515b1cc-72a3-4ba0-bcba-6e939421531d admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.431158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4515b1cc-72a3-4ba0-bcba-6e939421531d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.434248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4515b1cc-72a3-4ba0-bcba-6e939421531d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.458581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1234/2450] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:54 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:54.465777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4515b1cc-72a3-4ba0-bcba-6e939421531d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.483364 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4515b1cc-72a3-4ba0-bcba-6e939421531d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.490526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b137d9bf-41b2-4673-a9a1-d2ff80976da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.491684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b137d9bf-41b2-4673-a9a1-d2ff80976da4 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.492175 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b137d9bf-41b2-4673-a9a1-d2ff80976da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.492572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b137d9bf-41b2-4673-a9a1-d2ff80976da4 admin admin] PATH_INFO: `/v3/projects/6d26233fa2c14a5d9964e02c43b2e403/users/0e683da52fc94eddad32413aab85c56d/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.495621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b137d9bf-41b2-4673-a9a1-d2ff80976da4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d26233fa2c14a5d9964e02c43b2e403, user_id=0e683da52fc94eddad32413aab85c56d, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.498723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b137d9bf-41b2-4673-a9a1-d2ff80976da4 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.504061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c310efa-c33c-4800-a471-8a5625316c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.505107 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c310efa-c33c-4800-a471-8a5625316c46 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.505575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c310efa-c33c-4800-a471-8a5625316c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.506034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c310efa-c33c-4800-a471-8a5625316c46 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.507224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c310efa-c33c-4800-a471-8a5625316c46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.510118 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c310efa-c33c-4800-a471-8a5625316c46 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.512216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1235/2451] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:54 2022] PUT /identity/v3/projects/6d26233fa2c14a5d9964e02c43b2e403/users/0e683da52fc94eddad32413aab85c56d/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:54.525250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1217/2452] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:54 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:47:54.531988 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b137d9bf-41b2-4673-a9a1-d2ff80976da4 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.533113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf08caaf-7c57-4e96-acba-82ed20d42f9a None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.533510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf08caaf-7c57-4e96-acba-82ed20d42f9a None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.533986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf08caaf-7c57-4e96-acba-82ed20d42f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.571654 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-cf08caaf-7c57-4e96-acba-82ed20d42f9a None None] MFA Rules not processed for user `0e683da52fc94eddad32413aab85c56d`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:54.572913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-cf08caaf-7c57-4e96-acba-82ed20d42f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:54.631490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b137d9bf-41b2-4673-a9a1-d2ff80976da4 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.639788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.641445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.641911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.642397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] PATH_INFO: `/v3/projects/2047dcfb8e0240e2aee9991fd78904ce/users/6e5fa522caa04c92801a2295cfc7f729/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.645909 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] RBAC: Authorizing `identity:create_grant(project_id=2047dcfb8e0240e2aee9991fd78904ce, user_id=6e5fa522caa04c92801a2295cfc7f729, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.651515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.670622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1218/2453] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:54 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:54.672533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1236/2454] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:54 2022] PUT /identity/v3/projects/2047dcfb8e0240e2aee9991fd78904ce/users/6e5fa522caa04c92801a2295cfc7f729/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:54.688438 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.689085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a80c2-e9e4-499b-b6cc-3c7c41c8c518 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.689466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a80c2-e9e4-499b-b6cc-3c7c41c8c518 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.689855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a80c2-e9e4-499b-b6cc-3c7c41c8c518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.690670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:54.735357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-440a80c2-e9e4-499b-b6cc-3c7c41c8c518 None None] MFA Rules not processed for user `6e5fa522caa04c92801a2295cfc7f729`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:54.737281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-440a80c2-e9e4-499b-b6cc-3c7c41c8c518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:54.756515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:54.773001 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b847d320-a378-4ed7-b1a0-09ff42f58ebf admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.780341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb07ab6-32f8-4833-a131-f81407c10f23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.780341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb07ab6-32f8-4833-a131-f81407c10f23 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.780341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb07ab6-32f8-4833-a131-f81407c10f23 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.780341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb07ab6-32f8-4833-a131-f81407c10f23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.824704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:54.870220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb07ab6-32f8-4833-a131-f81407c10f23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.877023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1219/2455] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:54 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:54.877262 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb07ab6-32f8-4833-a131-f81407c10f23 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.887091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a80c2-e9e4-499b-b6cc-3c7c41c8c518 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.887404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1237/2456] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:54 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:54.888566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-440a80c2-e9e4-499b-b6cc-3c7c41c8c518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:54.902089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a80c2-e9e4-499b-b6cc-3c7c41c8c518 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.906681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c11d50-6037-474e-b2b2-5f427b6ca1e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.907330 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c11d50-6037-474e-b2b2-5f427b6ca1e6 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.907610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c11d50-6037-474e-b2b2-5f427b6ca1e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.907883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c11d50-6037-474e-b2b2-5f427b6ca1e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:54.919907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c11d50-6037-474e-b2b2-5f427b6ca1e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:54.924557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c11d50-6037-474e-b2b2-5f427b6ca1e6 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:54.942960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1220/2457] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:54 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:47:54.956796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb07ab6-32f8-4833-a131-f81407c10f23 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:54.971996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb07ab6-32f8-4833-a131-f81407c10f23 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:54.977641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:54.978586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:54.979118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:54.979605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:55.065392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:55.067922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:55.092724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1238/2458] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:54 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:55.114342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c11d50-6037-474e-b2b2-5f427b6ca1e6 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:55.148337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c11d50-6037-474e-b2b2-5f427b6ca1e6 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:55.155150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4220aab-c8d4-4bcf-acfa-69b85ec69113 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:55.156525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4220aab-c8d4-4bcf-acfa-69b85ec69113 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:55.157102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4220aab-c8d4-4bcf-acfa-69b85ec69113 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:55.157737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4220aab-c8d4-4bcf-acfa-69b85ec69113 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:55.258906 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4220aab-c8d4-4bcf-acfa-69b85ec69113 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:55.260498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4220aab-c8d4-4bcf-acfa-69b85ec69113 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:55.275554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1221/2459] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:55 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:55.484460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:55.485761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:55.630305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b8f676-1933-497b-b76b-16a849d08895 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:55.638228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:55.639603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:55.640117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:55.641087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] PATH_INFO: `/v3/users/77e911424c1b4e9cb7720370822b46d3` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:55.643130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=77e911424c1b4e9cb7720370822b46d3)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:55.646648 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:55.729976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77e911424c1b4e9cb7720370822b46d3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:55.739853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77e911424c1b4e9cb7720370822b46d3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:55.740696 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77e911424c1b4e9cb7720370822b46d3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:55.741398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77e911424c1b4e9cb7720370822b46d3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:55.757446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77e911424c1b4e9cb7720370822b46d3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:55.759791 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:55.828504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1239/2460] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:55 2022] DELETE /identity/v3/users/77e911424c1b4e9cb7720370822b46d3 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:55.832580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c69a268-6ac8-4f00-82fc-919536df8bef None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:55.832971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c69a268-6ac8-4f00-82fc-919536df8bef None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:55.833336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c69a268-6ac8-4f00-82fc-919536df8bef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:55.862648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7c69a268-6ac8-4f00-82fc-919536df8bef None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:55.863973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7c69a268-6ac8-4f00-82fc-919536df8bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:55.936196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1222/2461] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:55.947565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:55.949055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:56.035751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dbd505-47e2-4352-ae81-6970d271bb63 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:56.039719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3495728-ca5f-4cd7-90d5-95fe0851dd8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:56.040339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3495728-ca5f-4cd7-90d5-95fe0851dd8b service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:56.040624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3495728-ca5f-4cd7-90d5-95fe0851dd8b service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:56.040897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3495728-ca5f-4cd7-90d5-95fe0851dd8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:56.053014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3495728-ca5f-4cd7-90d5-95fe0851dd8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:56.054510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3495728-ca5f-4cd7-90d5-95fe0851dd8b service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:56.075552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1240/2462] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:56.394639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c69a268-6ac8-4f00-82fc-919536df8bef None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:56.396609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7c69a268-6ac8-4f00-82fc-919536df8bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:56.415525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c69a268-6ac8-4f00-82fc-919536df8bef None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:56.422454 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:56.423140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:56.423414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:56.423681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] PATH_INFO: `/v3/projects/346ed3941744457aa077b7b9fe51bf54` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:56.424825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=346ed3941744457aa077b7b9fe51bf54)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:56.426829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:56.482503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:56.499253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '346ed3941744457aa077b7b9fe51bf54', 'request_id': 'req-bcf6578c-607c-4509-a867-9a2a8624d3ca'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:56.506146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '346ed3941744457aa077b7b9fe51bf54', 'request_id': 'req-bcf6578c-607c-4509-a867-9a2a8624d3ca'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:56.524105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '346ed3941744457aa077b7b9fe51bf54', 'request_id': 'req-bcf6578c-607c-4509-a867-9a2a8624d3ca'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:56.538766 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] The token cache is being invalidate because project 346ed3941744457aa077b7b9fe51bf54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:47:56.539250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:56.540338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-bcf6578c-607c-4509-a867-9a2a8624d3ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:56.542792 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1223/2463] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:47:56 2022] DELETE /identity/v3/projects/346ed3941744457aa077b7b9fe51bf54 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:56.553555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c3264d-7c4d-4f3b-af7e-0cd5a8677a35 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:56.554064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c3264d-7c4d-4f3b-af7e-0cd5a8677a35 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:56.554520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c3264d-7c4d-4f3b-af7e-0cd5a8677a35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:56.579554 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-57c3264d-7c4d-4f3b-af7e-0cd5a8677a35 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:56.580798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-57c3264d-7c4d-4f3b-af7e-0cd5a8677a35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:56.668851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1241/2464] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:56.681953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4529f2d-e560-4488-85a9-9500915c138b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:56.682297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4529f2d-e560-4488-85a9-9500915c138b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:56.682555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4529f2d-e560-4488-85a9-9500915c138b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:56.709648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c4529f2d-e560-4488-85a9-9500915c138b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:56.712074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c4529f2d-e560-4488-85a9-9500915c138b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:56.725538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1224/2465] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:56.731941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c3264d-7c4d-4f3b-af7e-0cd5a8677a35 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:56.742211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c3264d-7c4d-4f3b-af7e-0cd5a8677a35 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:56.748993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:56.749781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:56.750101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:56.750447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:56.751214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:47:56.751553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:56.753347 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:56.774646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1242/2466] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:47:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:47:56.776209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4529f2d-e560-4488-85a9-9500915c138b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:56.795287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4529f2d-e560-4488-85a9-9500915c138b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:56.803619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cfeb8-274d-410c-afbd-a37a34c88872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:56.804665 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cfeb8-274d-410c-afbd-a37a34c88872 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:56.805117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cfeb8-274d-410c-afbd-a37a34c88872 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:56.805569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cfeb8-274d-410c-afbd-a37a34c88872 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:56.806835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04cfeb8-274d-410c-afbd-a37a34c88872 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:56.810486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04cfeb8-274d-410c-afbd-a37a34c88872 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:56.866884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1225/2467] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:56 2022] POST /identity/v3/projects => generated 365 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:56.872407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:56.893556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ae0b0b-a1f9-41d2-be23-cdcff4890c0b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:56.990777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee7e733-5e58-4421-a1ee-e9457192175b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:56.991958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee7e733-5e58-4421-a1ee-e9457192175b admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:56.992320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee7e733-5e58-4421-a1ee-e9457192175b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:56.993401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee7e733-5e58-4421-a1ee-e9457192175b admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:56.993961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee7e733-5e58-4421-a1ee-e9457192175b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:56.998077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee7e733-5e58-4421-a1ee-e9457192175b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.025476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:57.039143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1243/2468] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:56 2022] POST /identity/v3/users => generated 283 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:57.039295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cfeb8-274d-410c-afbd-a37a34c88872 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.052695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cfeb8-274d-410c-afbd-a37a34c88872 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.058439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7dc0b8-0043-41d5-878f-030dfe247a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.059175 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7dc0b8-0043-41d5-878f-030dfe247a97 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.059709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7dc0b8-0043-41d5-878f-030dfe247a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.059709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7dc0b8-0043-41d5-878f-030dfe247a97 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.060380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7dc0b8-0043-41d5-878f-030dfe247a97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.062636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7dc0b8-0043-41d5-878f-030dfe247a97 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.078795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1226/2469] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:57 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:57.086177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee7e733-5e58-4421-a1ee-e9457192175b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.101815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee7e733-5e58-4421-a1ee-e9457192175b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.111711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b82a1a-abdd-4795-9787-ee7439a3f314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.112913 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b82a1a-abdd-4795-9787-ee7439a3f314 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.113694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b82a1a-abdd-4795-9787-ee7439a3f314 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.114185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b82a1a-abdd-4795-9787-ee7439a3f314 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.115087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b82a1a-abdd-4795-9787-ee7439a3f314 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.118683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b82a1a-abdd-4795-9787-ee7439a3f314 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.144466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1244/2470] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:57 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:57.152644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7dc0b8-0043-41d5-878f-030dfe247a97 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.177561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7dc0b8-0043-41d5-878f-030dfe247a97 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.192188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90b3a66-9244-4322-912d-acf213923c4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.194186 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90b3a66-9244-4322-912d-acf213923c4c admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.194889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90b3a66-9244-4322-912d-acf213923c4c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.195517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90b3a66-9244-4322-912d-acf213923c4c admin admin] PATH_INFO: `/v3/projects/db1775710b53428ab05f5a0b2961f797/users/df0e96ca69c44f8382f4e4fc2028e5a4/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.239660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90b3a66-9244-4322-912d-acf213923c4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=db1775710b53428ab05f5a0b2961f797, user_id=df0e96ca69c44f8382f4e4fc2028e5a4, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.247770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90b3a66-9244-4322-912d-acf213923c4c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.279273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1227/2471] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:57 2022] PUT /identity/v3/projects/db1775710b53428ab05f5a0b2961f797/users/df0e96ca69c44f8382f4e4fc2028e5a4/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:47:57.299386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cf16fc-f0d4-4305-ab78-300b55b3f370 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.299885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cf16fc-f0d4-4305-ab78-300b55b3f370 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.300404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cf16fc-f0d4-4305-ab78-300b55b3f370 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.352784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-19cf16fc-f0d4-4305-ab78-300b55b3f370 None None] MFA Rules not processed for user `df0e96ca69c44f8382f4e4fc2028e5a4`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:57.354770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-19cf16fc-f0d4-4305-ab78-300b55b3f370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:57.453942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1245/2472] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:57 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:57.458986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90b3a66-9244-4322-912d-acf213923c4c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.539991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90b3a66-9244-4322-912d-acf213923c4c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.545808 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c03779-0919-4238-bc4b-8d60146c6e91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.547071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c03779-0919-4238-bc4b-8d60146c6e91 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.547071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c03779-0919-4238-bc4b-8d60146c6e91 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.547534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c03779-0919-4238-bc4b-8d60146c6e91 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.547977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c03779-0919-4238-bc4b-8d60146c6e91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.550559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c03779-0919-4238-bc4b-8d60146c6e91 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.587937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1228/2473] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:47:57 2022] POST /identity/v3/projects => generated 365 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:57.589961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cf16fc-f0d4-4305-ab78-300b55b3f370 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.606404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cf16fc-f0d4-4305-ab78-300b55b3f370 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.676328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691f718f-95df-406e-98ff-15902e2b0926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.677087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691f718f-95df-406e-98ff-15902e2b0926 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.677456 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691f718f-95df-406e-98ff-15902e2b0926 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.677819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691f718f-95df-406e-98ff-15902e2b0926 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.678581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691f718f-95df-406e-98ff-15902e2b0926 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.680961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691f718f-95df-406e-98ff-15902e2b0926 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.705704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1246/2474] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:47:57 2022] POST /identity/v3/users => generated 289 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:47:57.710199 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c03779-0919-4238-bc4b-8d60146c6e91 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.721905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c03779-0919-4238-bc4b-8d60146c6e91 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.728200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a314c52a-6803-4391-bd81-fd17d1d2c480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.729261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a314c52a-6803-4391-bd81-fd17d1d2c480 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.729541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a314c52a-6803-4391-bd81-fd17d1d2c480 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.729812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a314c52a-6803-4391-bd81-fd17d1d2c480 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.730482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a314c52a-6803-4391-bd81-fd17d1d2c480 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.732180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a314c52a-6803-4391-bd81-fd17d1d2c480 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.746335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1229/2475] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:57 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:57.751335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691f718f-95df-406e-98ff-15902e2b0926 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.762818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691f718f-95df-406e-98ff-15902e2b0926 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.768187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5448cc9e-cfa4-43c7-8a4c-c13333b7b06a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.768911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5448cc9e-cfa4-43c7-8a4c-c13333b7b06a admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.769221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5448cc9e-cfa4-43c7-8a4c-c13333b7b06a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.769504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5448cc9e-cfa4-43c7-8a4c-c13333b7b06a admin admin] PATH_INFO: `/v3/projects/3a4a141581c54f4eb8f26932fe4ba34b/users/629a06b61c454afb8f444163f20eaef9/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.790943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5448cc9e-cfa4-43c7-8a4c-c13333b7b06a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a4a141581c54f4eb8f26932fe4ba34b, user_id=629a06b61c454afb8f444163f20eaef9, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.793672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5448cc9e-cfa4-43c7-8a4c-c13333b7b06a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.815686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1247/2476] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:57 2022] PUT /identity/v3/projects/3a4a141581c54f4eb8f26932fe4ba34b/users/629a06b61c454afb8f444163f20eaef9/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:57.821050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a314c52a-6803-4391-bd81-fd17d1d2c480 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.838240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a314c52a-6803-4391-bd81-fd17d1d2c480 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.881107 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:57.924973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbda007-7a6b-443b-8f91-a17e2870fcf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.925726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbda007-7a6b-443b-8f91-a17e2870fcf3 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.926010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbda007-7a6b-443b-8f91-a17e2870fcf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.926288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbda007-7a6b-443b-8f91-a17e2870fcf3 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.927009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbda007-7a6b-443b-8f91-a17e2870fcf3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.928765 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbda007-7a6b-443b-8f91-a17e2870fcf3 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:57.942874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1230/2477] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:47:57 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:47:57.948678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5448cc9e-cfa4-43c7-8a4c-c13333b7b06a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:57.965834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5448cc9e-cfa4-43c7-8a4c-c13333b7b06a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:57.976463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:57.977195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:57.977481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:57.977779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] PATH_INFO: `/v3/projects/3a4a141581c54f4eb8f26932fe4ba34b/users/629a06b61c454afb8f444163f20eaef9/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:57.980119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a4a141581c54f4eb8f26932fe4ba34b, user_id=629a06b61c454afb8f444163f20eaef9, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:57.987462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:58.001298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1248/2478] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:47:57 2022] PUT /identity/v3/projects/3a4a141581c54f4eb8f26932fe4ba34b/users/629a06b61c454afb8f444163f20eaef9/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:58.013290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d02509-3176-4916-95d7-c60a9693692f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:58.013568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d02509-3176-4916-95d7-c60a9693692f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:58.013840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d02509-3176-4916-95d7-c60a9693692f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:58.053986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b6d02509-3176-4916-95d7-c60a9693692f None None] MFA Rules not processed for user `629a06b61c454afb8f444163f20eaef9`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:58.055926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b6d02509-3176-4916-95d7-c60a9693692f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:58.156675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1231/2479] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:58 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:58.170592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:58.172472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:58.218357 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:58.249064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105810f-c667-47cf-b88b-cbdbd309514f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:58.253976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03561233-4d5e-4196-aeac-2cd386f22c53 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:58.254603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03561233-4d5e-4196-aeac-2cd386f22c53 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:58.255094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03561233-4d5e-4196-aeac-2cd386f22c53 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:58.255479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03561233-4d5e-4196-aeac-2cd386f22c53 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:58.268016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03561233-4d5e-4196-aeac-2cd386f22c53 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:58.269525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03561233-4d5e-4196-aeac-2cd386f22c53 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:58.285893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1249/2480] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:58 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:58.293112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d02509-3176-4916-95d7-c60a9693692f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:58.294632 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b6d02509-3176-4916-95d7-c60a9693692f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:58.386399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d02509-3176-4916-95d7-c60a9693692f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:58.396453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:58.398211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:58.398761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:58.399218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] PATH_INFO: `/v3/users/1aee5fe10f074ec18305b843233f475d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:58.401738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=1aee5fe10f074ec18305b843233f475d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:58.405222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:58.536238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1aee5fe10f074ec18305b843233f475d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:58.537592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1aee5fe10f074ec18305b843233f475d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:58.550606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:58.554663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1aee5fe10f074ec18305b843233f475d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:58.576524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1aee5fe10f074ec18305b843233f475d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:58.595236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1aee5fe10f074ec18305b843233f475d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:58.609387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1232/2481] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:58 2022] DELETE /identity/v3/users/1aee5fe10f074ec18305b843233f475d => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:58.621790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49390cf-7127-4e7d-9724-b79a4ce0f5c6 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:58.622548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49390cf-7127-4e7d-9724-b79a4ce0f5c6 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:58.622548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49390cf-7127-4e7d-9724-b79a4ce0f5c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:58.658993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d49390cf-7127-4e7d-9724-b79a4ce0f5c6 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:58.660445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d49390cf-7127-4e7d-9724-b79a4ce0f5c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:58.762632 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1250/2482] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:58.778359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:58.782000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:58.906784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a09896-0413-42b6-908c-b9ed15aa5a2f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:58.912377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf8be1d-b5d5-452d-a807-25b857b5aab9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:58.913437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf8be1d-b5d5-452d-a807-25b857b5aab9 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:58.914843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf8be1d-b5d5-452d-a807-25b857b5aab9 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:58.914843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf8be1d-b5d5-452d-a807-25b857b5aab9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:58.932965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf8be1d-b5d5-452d-a807-25b857b5aab9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:58.935143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf8be1d-b5d5-452d-a807-25b857b5aab9 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:58.951211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1233/2483] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:59.065254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49390cf-7127-4e7d-9724-b79a4ce0f5c6 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:59.068187 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d49390cf-7127-4e7d-9724-b79a4ce0f5c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:59.093915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49390cf-7127-4e7d-9724-b79a4ce0f5c6 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:59.100458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:59.101453 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:59.101743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:59.102127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] PATH_INFO: `/v3/users/df0e96ca69c44f8382f4e4fc2028e5a4` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:59.106776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df0e96ca69c44f8382f4e4fc2028e5a4)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:59.106776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:59.192568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df0e96ca69c44f8382f4e4fc2028e5a4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.226785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df0e96ca69c44f8382f4e4fc2028e5a4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.227694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df0e96ca69c44f8382f4e4fc2028e5a4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.228819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df0e96ca69c44f8382f4e4fc2028e5a4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.242491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df0e96ca69c44f8382f4e4fc2028e5a4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.277471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1251/2484] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:59 2022] DELETE /identity/v3/users/df0e96ca69c44f8382f4e4fc2028e5a4 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:59.280262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075464ae-18a8-4c65-ba0b-67b2b2a2f361 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:59.281618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075464ae-18a8-4c65-ba0b-67b2b2a2f361 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:59.281618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075464ae-18a8-4c65-ba0b-67b2b2a2f361 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:59.329332 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-075464ae-18a8-4c65-ba0b-67b2b2a2f361 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:59.331757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-075464ae-18a8-4c65-ba0b-67b2b2a2f361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:59.408337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:59.410850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:59.470605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1234/2485] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:47:59.491471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075464ae-18a8-4c65-ba0b-67b2b2a2f361 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:59.495721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-075464ae-18a8-4c65-ba0b-67b2b2a2f361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:59.496737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:59.528044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb9ca35-ee08-40c6-8ea6-e1ada9c30326 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:59.537470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:59.538283 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:59.538641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:59.539038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] PATH_INFO: `/v3/users/5e83d357404b47c28e4c777998cbfdd2` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:59.540290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e83d357404b47c28e4c777998cbfdd2)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:59.543783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:59.622375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e83d357404b47c28e4c777998cbfdd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.632955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e83d357404b47c28e4c777998cbfdd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.633896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e83d357404b47c28e4c777998cbfdd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.634940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e83d357404b47c28e4c777998cbfdd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.636851 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075464ae-18a8-4c65-ba0b-67b2b2a2f361 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:59.642202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5f4606-da69-401c-a2cd-efb46cfca6b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:59.643127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f4606-da69-401c-a2cd-efb46cfca6b0 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:59.643540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f4606-da69-401c-a2cd-efb46cfca6b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:59.644023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f4606-da69-401c-a2cd-efb46cfca6b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:59.647018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ce5f4606-da69-401c-a2cd-efb46cfca6b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:59.647382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e83d357404b47c28e4c777998cbfdd2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.668433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5f4606-da69-401c-a2cd-efb46cfca6b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:59.670477 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5f4606-da69-401c-a2cd-efb46cfca6b0 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:59.673609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1252/2486] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:59 2022] DELETE /identity/v3/users/5e83d357404b47c28e4c777998cbfdd2 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:59.675242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:59.677353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:59.687971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1235/2487] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:47:59.689273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc785eeb-4a6d-47f1-898a-40a159a5803e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:59.689709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc785eeb-4a6d-47f1-898a-40a159a5803e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:59.690047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc785eeb-4a6d-47f1-898a-40a159a5803e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:59.714013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-cc785eeb-4a6d-47f1-898a-40a159a5803e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:47:59.715112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-cc785eeb-4a6d-47f1-898a-40a159a5803e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:59.733139 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:59.813640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba01775-e166-41e0-a16e-ca85cdb24864 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:59.823999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:47:59.824529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:47:59.824833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:47:59.825113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] PATH_INFO: `/v3/projects/c5ccca60312e404e97077965d5ec0ee5` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:47:59.826346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5ccca60312e404e97077965d5ec0ee5)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:47:59.828543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:47:59.833467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1236/2488] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:47:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:47:59.843842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc785eeb-4a6d-47f1-898a-40a159a5803e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:59.845570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-cc785eeb-4a6d-47f1-898a-40a159a5803e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:47:59.914897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5ccca60312e404e97077965d5ec0ee5', 'request_id': 'req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.929104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5ccca60312e404e97077965d5ec0ee5', 'request_id': 'req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.941776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5ccca60312e404e97077965d5ec0ee5', 'request_id': 'req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.948878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:47:59.967787 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc785eeb-4a6d-47f1-898a-40a159a5803e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:47:59.971583 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] The token cache is being invalidate because project c5ccca60312e404e97077965d5ec0ee5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:47:59.972180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.975640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:47:59.979457 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1253/2489] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:47:59 2022] DELETE /identity/v3/projects/c5ccca60312e404e97077965d5ec0ee5 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:47:59.983556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:47:59.989803 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:00.069810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:00.070976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:00.071697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:00.072213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:00.076334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:00.111266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8624e0fc-6f5d-4178-b794-9e3a5b5bb254 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:00.118076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:00.119592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:00.119592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:00.119592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] PATH_INFO: `/v3/users/6e5fa522caa04c92801a2295cfc7f729` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:00.120942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e5fa522caa04c92801a2295cfc7f729)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:00.123901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:00.204478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:00.208016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:00.215107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e5fa522caa04c92801a2295cfc7f729'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.228011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1237/2490] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:47:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:00.229472 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:00.235153 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:00.247006 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e5fa522caa04c92801a2295cfc7f729'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.248275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e5fa522caa04c92801a2295cfc7f729'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.249575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e5fa522caa04c92801a2295cfc7f729'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.265303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f2d7d5-f209-4ec1-bc5b-afc53bb2db42 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:00.269720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e5fa522caa04c92801a2295cfc7f729'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.273919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:00.275305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:00.275784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:00.276266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] PATH_INFO: `/v3/projects/db1775710b53428ab05f5a0b2961f797` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:00.278427 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db1775710b53428ab05f5a0b2961f797)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:00.282749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:00.317035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1254/2491] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:47:59 2022] DELETE /identity/v3/users/6e5fa522caa04c92801a2295cfc7f729 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:00.329121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:00.374309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db1775710b53428ab05f5a0b2961f797', 'request_id': 'req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.387599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db1775710b53428ab05f5a0b2961f797', 'request_id': 'req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.401024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db1775710b53428ab05f5a0b2961f797', 'request_id': 'req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.424205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] The token cache is being invalidate because project db1775710b53428ab05f5a0b2961f797 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:00.424694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.425854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.428647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1238/2492] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:00 2022] DELETE /identity/v3/projects/db1775710b53428ab05f5a0b2961f797 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:00.433596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:00.436113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:00.587936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398c926d-b4bb-4e86-900e-25f85c13a25d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:00.622872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:00.713830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:00.715232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:00.716207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:00.716846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] PATH_INFO: `/v3/users/629a06b61c454afb8f444163f20eaef9` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:00.719345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=629a06b61c454afb8f444163f20eaef9)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:00.722848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:00.770576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:00.773441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:00.799061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7425c8-0fd2-4beb-805a-bc244f0ebc96 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:00.807770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:00.808777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:00.809095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:00.809480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] PATH_INFO: `/v3/projects/3e4072234bbc4a03b793aa756930330b` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:00.816067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e4072234bbc4a03b793aa756930330b)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:00.825969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '629a06b61c454afb8f444163f20eaef9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.836795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:00.837097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '629a06b61c454afb8f444163f20eaef9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.838039 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '629a06b61c454afb8f444163f20eaef9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.838949 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '629a06b61c454afb8f444163f20eaef9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.852930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '629a06b61c454afb8f444163f20eaef9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.891524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1255/2493] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:00 2022] DELETE /identity/v3/users/629a06b61c454afb8f444163f20eaef9 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:00.895377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:00.911450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:00.936062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e4072234bbc4a03b793aa756930330b', 'request_id': 'req-f56234aa-4453-43bb-869d-ec093633f965'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.955704 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e4072234bbc4a03b793aa756930330b', 'request_id': 'req-f56234aa-4453-43bb-869d-ec093633f965'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.967882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e4072234bbc4a03b793aa756930330b', 'request_id': 'req-f56234aa-4453-43bb-869d-ec093633f965'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.988332 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] The token cache is being invalidate because project 3e4072234bbc4a03b793aa756930330b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:00.988874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.990308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:00.993085 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1239/2494] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:00 2022] DELETE /identity/v3/projects/3e4072234bbc4a03b793aa756930330b => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:01.000648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:01.003115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:01.039186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32883ab-3b39-4d1b-ab93-ef89c47334b9 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:01.115788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56234aa-4453-43bb-869d-ec093633f965 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:01.123416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:01.127219 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.127511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.127955 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] PATH_INFO: `/v3/users/0e683da52fc94eddad32413aab85c56d` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.129813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e683da52fc94eddad32413aab85c56d)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:01.131825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:01.139473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:01.166920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:01.167962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.168517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.168936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] PATH_INFO: `/v3/projects/2047dcfb8e0240e2aee9991fd78904ce` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.171267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2047dcfb8e0240e2aee9991fd78904ce)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:01.174183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:01.188205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:01.211709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e683da52fc94eddad32413aab85c56d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.213453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e683da52fc94eddad32413aab85c56d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.227814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e683da52fc94eddad32413aab85c56d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.239930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e683da52fc94eddad32413aab85c56d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.262060 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2047dcfb8e0240e2aee9991fd78904ce', 'request_id': 'req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.271143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2047dcfb8e0240e2aee9991fd78904ce', 'request_id': 'req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.272877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-fb63d329-247e-4d5e-affb-f2331b3e7bcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e683da52fc94eddad32413aab85c56d'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.279458 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2047dcfb8e0240e2aee9991fd78904ce', 'request_id': 'req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.286433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1240/2495] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:00 2022] DELETE /identity/v3/users/0e683da52fc94eddad32413aab85c56d => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:01.315853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] The token cache is being invalidate because project 2047dcfb8e0240e2aee9991fd78904ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:01.316185 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.317966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.324266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1256/2496] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:00 2022] DELETE /identity/v3/projects/2047dcfb8e0240e2aee9991fd78904ce => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:01.327996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d13e0d-a594-45bd-af90-aa5810ae253f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.328803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d13e0d-a594-45bd-af90-aa5810ae253f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.329297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d13e0d-a594-45bd-af90-aa5810ae253f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.338642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:01.342501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:01.392060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-55d13e0d-a594-45bd-af90-aa5810ae253f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:01.395115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-55d13e0d-a594-45bd-af90-aa5810ae253f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:01.475765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8ff472-a173-4b9f-a1a6-3a3ce688cab6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:01.548816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1241/2497] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:01.557556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8295f91-6c9b-48f5-9958-479fd222ec4b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.557874 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8295f91-6c9b-48f5-9958-479fd222ec4b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.558163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8295f91-6c9b-48f5-9958-479fd222ec4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.581177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d8295f91-6c9b-48f5-9958-479fd222ec4b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:01.582442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d8295f91-6c9b-48f5-9958-479fd222ec4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:01.592012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:01.593189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.593730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.594238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] PATH_INFO: `/v3/projects/3a4a141581c54f4eb8f26932fe4ba34b` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.595718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a4a141581c54f4eb8f26932fe4ba34b)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:01.598809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1242/2498] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:01.599621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:01.600206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8295f91-6c9b-48f5-9958-479fd222ec4b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:01.618184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8295f91-6c9b-48f5-9958-479fd222ec4b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:01.627137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:01.627946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.628296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.628753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.630369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:01.630369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:01.633762 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:01.649640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:01.657400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1243/2499] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:01.663049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:01.672614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a4a141581c54f4eb8f26932fe4ba34b', 'request_id': 'req-351f4c25-e0a2-404d-a952-969ec5ac0b51'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.676680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c90b186-2b6e-428c-aa43-5319f377f3c9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:01.679898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a4a141581c54f4eb8f26932fe4ba34b', 'request_id': 'req-351f4c25-e0a2-404d-a952-969ec5ac0b51'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.687853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a4a141581c54f4eb8f26932fe4ba34b', 'request_id': 'req-351f4c25-e0a2-404d-a952-969ec5ac0b51'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.714054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] The token cache is being invalidate because project 3a4a141581c54f4eb8f26932fe4ba34b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:01.714513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.717110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-351f4c25-e0a2-404d-a952-969ec5ac0b51 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:01.721430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1257/2500] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:01 2022] DELETE /identity/v3/projects/3a4a141581c54f4eb8f26932fe4ba34b => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:01.728807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fefab61-c904-4f2b-929c-0638abe0f1cd None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.729302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fefab61-c904-4f2b-929c-0638abe0f1cd None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.729715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fefab61-c904-4f2b-929c-0638abe0f1cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.761265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8fefab61-c904-4f2b-929c-0638abe0f1cd None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:01.763654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8fefab61-c904-4f2b-929c-0638abe0f1cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:01.801663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:01.802414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.802681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.803064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.803777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:01.805810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:01.846926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1244/2501] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:01 2022] POST /identity/v3/projects => generated 371 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:01.846926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:01.850146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:01.873033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1258/2502] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:01.873803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:01.874023 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fefab61-c904-4f2b-929c-0638abe0f1cd None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:01.875872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8fefab61-c904-4f2b-929c-0638abe0f1cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:01.892614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fefab61-c904-4f2b-929c-0638abe0f1cd None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:01.960438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691e060-a0bc-4768-94e9-353b61b686fe admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:01.966161 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:01.966858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.967145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.967424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] PATH_INFO: `/v3/projects/6d26233fa2c14a5d9964e02c43b2e403` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.969442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d26233fa2c14a5d9964e02c43b2e403)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:01.971419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:01.984406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c612e94b-8362-4411-b033-5fce159f9102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:01.985285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c612e94b-8362-4411-b033-5fce159f9102 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:01.985653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c612e94b-8362-4411-b033-5fce159f9102 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:01.985945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c612e94b-8362-4411-b033-5fce159f9102 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:01.986669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c612e94b-8362-4411-b033-5fce159f9102 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:01.989592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c612e94b-8362-4411-b033-5fce159f9102 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.021387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1259/2503] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:01 2022] POST /identity/v3/users => generated 286 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:02.022764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5309b18-8952-4170-97c9-f2b263f3e6a5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.023133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5309b18-8952-4170-97c9-f2b263f3e6a5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.023411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5309b18-8952-4170-97c9-f2b263f3e6a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.050020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d26233fa2c14a5d9964e02c43b2e403', 'request_id': 'req-59ce9ee5-293f-41f0-bd58-2189b37c22b6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:02.053397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-a5309b18-8952-4170-97c9-f2b263f3e6a5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:02.055105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a5309b18-8952-4170-97c9-f2b263f3e6a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:02.061215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d26233fa2c14a5d9964e02c43b2e403', 'request_id': 'req-59ce9ee5-293f-41f0-bd58-2189b37c22b6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:02.073475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d26233fa2c14a5d9964e02c43b2e403', 'request_id': 'req-59ce9ee5-293f-41f0-bd58-2189b37c22b6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:02.091082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] The token cache is being invalidate because project 6d26233fa2c14a5d9964e02c43b2e403 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:02.091693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:02.092626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:02.095239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1245/2504] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:01 2022] DELETE /identity/v3/projects/6d26233fa2c14a5d9964e02c43b2e403 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:02.095692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.097056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:02.124937 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:02.157420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1260/2505] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:02.159340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de42eb94-a2ed-422a-bca1-1c89bdea49c9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.159795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de42eb94-a2ed-422a-bca1-1c89bdea49c9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.160343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de42eb94-a2ed-422a-bca1-1c89bdea49c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.189005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ce9ee5-293f-41f0-bd58-2189b37c22b6 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.189661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-de42eb94-a2ed-422a-bca1-1c89bdea49c9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:02.191028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-de42eb94-a2ed-422a-bca1-1c89bdea49c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:02.197009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31b1abc-fa0b-4d07-a821-f3e2c518b6be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.197721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31b1abc-fa0b-4d07-a821-f3e2c518b6be admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.198075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31b1abc-fa0b-4d07-a821-f3e2c518b6be admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.198533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31b1abc-fa0b-4d07-a821-f3e2c518b6be admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.199636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31b1abc-fa0b-4d07-a821-f3e2c518b6be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.201359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31b1abc-fa0b-4d07-a821-f3e2c518b6be admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.211477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1261/2506] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:02.211477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de42eb94-a2ed-422a-bca1-1c89bdea49c9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.213228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1246/2507] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:02 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:02.218371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514153a6-c20e-449f-b68e-9e46e2c6e0c7 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.218371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514153a6-c20e-449f-b68e-9e46e2c6e0c7 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.218371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514153a6-c20e-449f-b68e-9e46e2c6e0c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.223402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de42eb94-a2ed-422a-bca1-1c89bdea49c9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.231958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.232746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.233242 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.233577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.234871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:02.235635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.238450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.256637 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-514153a6-c20e-449f-b68e-9e46e2c6e0c7 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:02.258063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-514153a6-c20e-449f-b68e-9e46e2c6e0c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:02.258446 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1262/2508] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:02.259079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.272602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1247/2509] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:02.273605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514153a6-c20e-449f-b68e-9e46e2c6e0c7 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.279495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75505a7-3188-407a-9698-921d1bcde90d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.286268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98862d76-87b1-4573-ad42-a19ec5295065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.287373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98862d76-87b1-4573-ad42-a19ec5295065 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.287753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98862d76-87b1-4573-ad42-a19ec5295065 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.288116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98862d76-87b1-4573-ad42-a19ec5295065 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.288979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98862d76-87b1-4573-ad42-a19ec5295065 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.290862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98862d76-87b1-4573-ad42-a19ec5295065 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.291023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514153a6-c20e-449f-b68e-9e46e2c6e0c7 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.297022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b69ea5c-5377-4493-9a8d-a41be5b0b0c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.297747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69ea5c-5377-4493-9a8d-a41be5b0b0c1 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.298066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69ea5c-5377-4493-9a8d-a41be5b0b0c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.298288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69ea5c-5377-4493-9a8d-a41be5b0b0c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.299097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b69ea5c-5377-4493-9a8d-a41be5b0b0c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.301284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b69ea5c-5377-4493-9a8d-a41be5b0b0c1 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.316104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1263/2510] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:02 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:02.317073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98862d76-87b1-4573-ad42-a19ec5295065 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.340052 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98862d76-87b1-4573-ad42-a19ec5295065 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.347345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1248/2511] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:02 2022] POST /identity/v3/projects => generated 367 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:02.347345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b69ea5c-5377-4493-9a8d-a41be5b0b0c1 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.370416 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:02.436866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.437488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.437884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.438179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.439303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:02.439796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.441628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.449774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b69ea5c-5377-4493-9a8d-a41be5b0b0c1 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.457969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19a2b93-95a2-456e-a1e3-dff021726f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.458666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a2b93-95a2-456e-a1e3-dff021726f87 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.458960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a2b93-95a2-456e-a1e3-dff021726f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.459231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a2b93-95a2-456e-a1e3-dff021726f87 admin admin] PATH_INFO: `/v3/projects/91d84016860e45abb691c5163177775d/users/f56e532fa8fb4f4ca56f3b4a0bd7d574/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.463722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1264/2512] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:02.470653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.480904 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19a2b93-95a2-456e-a1e3-dff021726f87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91d84016860e45abb691c5163177775d, user_id=f56e532fa8fb4f4ca56f3b4a0bd7d574, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.485104 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19a2b93-95a2-456e-a1e3-dff021726f87 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.487472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90039fa0-90d8-49c3-b370-34ae6cc3e32a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.503143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf0840f-df34-4e7c-b29a-ef674dfa5cdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.504120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf0840f-df34-4e7c-b29a-ef674dfa5cdf admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.504531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf0840f-df34-4e7c-b29a-ef674dfa5cdf admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.504893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf0840f-df34-4e7c-b29a-ef674dfa5cdf admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.506108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf0840f-df34-4e7c-b29a-ef674dfa5cdf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.509750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf0840f-df34-4e7c-b29a-ef674dfa5cdf admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.513138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1249/2513] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:02 2022] PUT /identity/v3/projects/91d84016860e45abb691c5163177775d/users/f56e532fa8fb4f4ca56f3b4a0bd7d574/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:02.514020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19a2b93-95a2-456e-a1e3-dff021726f87 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.544992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1265/2514] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:02 2022] POST /identity/v3/users => generated 284 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:02.550656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047809c4-94a0-47cd-bb09-0c1416e89b3f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.551169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047809c4-94a0-47cd-bb09-0c1416e89b3f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.551486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047809c4-94a0-47cd-bb09-0c1416e89b3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.592794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-047809c4-94a0-47cd-bb09-0c1416e89b3f None None] MFA Rules not processed for user `f56e532fa8fb4f4ca56f3b4a0bd7d574`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:02.595556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-047809c4-94a0-47cd-bb09-0c1416e89b3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:02.607789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19a2b93-95a2-456e-a1e3-dff021726f87 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.613282 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1137b8-cc88-49b6-9381-d6fb362ffafd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.613988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1137b8-cc88-49b6-9381-d6fb362ffafd admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.614501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1137b8-cc88-49b6-9381-d6fb362ffafd admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.614854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1137b8-cc88-49b6-9381-d6fb362ffafd admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.615519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1137b8-cc88-49b6-9381-d6fb362ffafd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.617420 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1137b8-cc88-49b6-9381-d6fb362ffafd admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.656793 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1250/2515] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:02 2022] POST /identity/v3/projects => generated 387 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:02.658272 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1137b8-cc88-49b6-9381-d6fb362ffafd admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.692724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1266/2516] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:02 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:02.692724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047809c4-94a0-47cd-bb09-0c1416e89b3f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.711881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047809c4-94a0-47cd-bb09-0c1416e89b3f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.743369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:02.765954 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1137b8-cc88-49b6-9381-d6fb362ffafd admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.774902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.775723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.776117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.776460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.778073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.779757 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.797756 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1251/2517] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:02 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:02.799575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.802811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:02.808679 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bb96a1-b7e6-4aa5-924e-b08b8b20f3a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.809383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb96a1-b7e6-4aa5-924e-b08b8b20f3a0 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.809726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb96a1-b7e6-4aa5-924e-b08b8b20f3a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.810149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb96a1-b7e6-4aa5-924e-b08b8b20f3a0 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.811139 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bb96a1-b7e6-4aa5-924e-b08b8b20f3a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.813548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bb96a1-b7e6-4aa5-924e-b08b8b20f3a0 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.841585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:02.844550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1267/2518] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:02 2022] POST /identity/v3/users => generated 294 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:02.845311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bb96a1-b7e6-4aa5-924e-b08b8b20f3a0 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.862278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bb96a1-b7e6-4aa5-924e-b08b8b20f3a0 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.875486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d4ed55-69e1-4642-b380-3ccd6e89aaa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.876153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d4ed55-69e1-4642-b380-3ccd6e89aaa8 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.876586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d4ed55-69e1-4642-b380-3ccd6e89aaa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.876869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d4ed55-69e1-4642-b380-3ccd6e89aaa8 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.877543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d4ed55-69e1-4642-b380-3ccd6e89aaa8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.879396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d4ed55-69e1-4642-b380-3ccd6e89aaa8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.895818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1268/2519] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:02 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:02.895818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d4ed55-69e1-4642-b380-3ccd6e89aaa8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.901947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fea50d-0f7c-444b-ad29-fa35ae5c45e7 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.903959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d4ed55-69e1-4642-b380-3ccd6e89aaa8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.911871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddcf0e1-09a5-4285-a84b-59cf87197356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.912206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b69529a-6f2a-4077-8d3e-443b3fd5fb74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.913325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddcf0e1-09a5-4285-a84b-59cf87197356 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.913583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69529a-6f2a-4077-8d3e-443b3fd5fb74 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.913762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddcf0e1-09a5-4285-a84b-59cf87197356 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.913939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69529a-6f2a-4077-8d3e-443b3fd5fb74 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.914114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddcf0e1-09a5-4285-a84b-59cf87197356 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.914275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69529a-6f2a-4077-8d3e-443b3fd5fb74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.914708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddcf0e1-09a5-4285-a84b-59cf87197356 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.916625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddcf0e1-09a5-4285-a84b-59cf87197356 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:02.930272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1269/2520] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:02 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:02.931182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddcf0e1-09a5-4285-a84b-59cf87197356 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:02.950851 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddcf0e1-09a5-4285-a84b-59cf87197356 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:02.959067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472052a4-4421-4e14-9605-aa9f9a74dc3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:02.960266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472052a4-4421-4e14-9605-aa9f9a74dc3e admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:02.960774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472052a4-4421-4e14-9605-aa9f9a74dc3e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:02.961266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472052a4-4421-4e14-9605-aa9f9a74dc3e admin admin] PATH_INFO: `/v3/projects/d21f41e8a9b6429a8871af251eb9df68/users/2498e216739545418885a7c6e03a773a/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:02.983415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472052a4-4421-4e14-9605-aa9f9a74dc3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d21f41e8a9b6429a8871af251eb9df68, user_id=2498e216739545418885a7c6e03a773a, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:02.987080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472052a4-4421-4e14-9605-aa9f9a74dc3e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.005696 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1270/2521] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:02 2022] PUT /identity/v3/projects/d21f41e8a9b6429a8871af251eb9df68/users/2498e216739545418885a7c6e03a773a/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:03.006898 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472052a4-4421-4e14-9605-aa9f9a74dc3e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.020022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b69529a-6f2a-4077-8d3e-443b3fd5fb74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.022679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b69529a-6f2a-4077-8d3e-443b3fd5fb74 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.039164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1252/2522] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:02 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:03.041508 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599769a0-624f-499c-a69f-a826f1744be5 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.041973 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599769a0-624f-499c-a69f-a826f1744be5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.042367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599769a0-624f-499c-a69f-a826f1744be5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.087115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-599769a0-624f-499c-a69f-a826f1744be5 None None] MFA Rules not processed for user `2498e216739545418885a7c6e03a773a`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:03.089259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-599769a0-624f-499c-a69f-a826f1744be5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:03.115936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472052a4-4421-4e14-9605-aa9f9a74dc3e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.123372 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeea1718-aaa6-40a3-a277-04216d348caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.124362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeea1718-aaa6-40a3-a277-04216d348caa admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.124842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeea1718-aaa6-40a3-a277-04216d348caa admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.125279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeea1718-aaa6-40a3-a277-04216d348caa admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.126451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeea1718-aaa6-40a3-a277-04216d348caa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.128718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeea1718-aaa6-40a3-a277-04216d348caa admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.143736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1271/2523] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:03 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:03.145597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeea1718-aaa6-40a3-a277-04216d348caa admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.162046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeea1718-aaa6-40a3-a277-04216d348caa admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.175637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.176602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.177106 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.177539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] PATH_INFO: `/v3/users/f56e532fa8fb4f4ca56f3b4a0bd7d574` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.179689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f56e532fa8fb4f4ca56f3b4a0bd7d574)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.183182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.191645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1253/2524] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:03 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:03.192873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599769a0-624f-499c-a69f-a826f1744be5 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.210931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599769a0-624f-499c-a69f-a826f1744be5 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.221149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b2e183-76fb-43d2-8f78-f7ac9fefea72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.221820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b2e183-76fb-43d2-8f78-f7ac9fefea72 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.222097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b2e183-76fb-43d2-8f78-f7ac9fefea72 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.222395 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b2e183-76fb-43d2-8f78-f7ac9fefea72 admin admin] PATH_INFO: `/v3/projects/ac895a47589d48b99cb6f3964579c121/users/ae5fbe31ee71437cbf18bfcfe451d19e/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.240320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b2e183-76fb-43d2-8f78-f7ac9fefea72 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac895a47589d48b99cb6f3964579c121, user_id=ae5fbe31ee71437cbf18bfcfe451d19e, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.243223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b2e183-76fb-43d2-8f78-f7ac9fefea72 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.254307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f56e532fa8fb4f4ca56f3b4a0bd7d574'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:03.263539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1254/2525] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:03 2022] PUT /identity/v3/projects/ac895a47589d48b99cb6f3964579c121/users/ae5fbe31ee71437cbf18bfcfe451d19e/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:03.264685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b2e183-76fb-43d2-8f78-f7ac9fefea72 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.282157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f56e532fa8fb4f4ca56f3b4a0bd7d574'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:03.283192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f56e532fa8fb4f4ca56f3b4a0bd7d574'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:03.283881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f56e532fa8fb4f4ca56f3b4a0bd7d574'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:03.286209 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:03.295115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-6182158c-bf3d-4cb9-bedc-2aa9ec2452c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f56e532fa8fb4f4ca56f3b4a0bd7d574'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:03.334999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1272/2526] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:03 2022] DELETE /identity/v3/users/f56e532fa8fb4f4ca56f3b4a0bd7d574 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:03.339387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302940f9-b4e7-4046-a413-d61d0f756cea None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.339881 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302940f9-b4e7-4046-a413-d61d0f756cea None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.340025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302940f9-b4e7-4046-a413-d61d0f756cea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.342148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:03.373763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b2e183-76fb-43d2-8f78-f7ac9fefea72 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.384711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-302940f9-b4e7-4046-a413-d61d0f756cea None None] MFA Rules not processed for user `ae5fbe31ee71437cbf18bfcfe451d19e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:03.386423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-302940f9-b4e7-4046-a413-d61d0f756cea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:03.467816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.468987 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.469495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.469995 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.471109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.474971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.488340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1273/2527] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:03 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:03.489805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fb7ed6-b807-4eb2-8d31-c55075b1cc4f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.490204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fb7ed6-b807-4eb2-8d31-c55075b1cc4f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.490638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fb7ed6-b807-4eb2-8d31-c55075b1cc4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.521424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-29fb7ed6-b807-4eb2-8d31-c55075b1cc4f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:03.522700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-29fb7ed6-b807-4eb2-8d31-c55075b1cc4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:03.523909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1255/2528] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:03 2022] POST /identity/v3/projects => generated 369 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:03.525393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.528253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:03.605575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1274/2529] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:03.606423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fb7ed6-b807-4eb2-8d31-c55075b1cc4f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.613362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-29fb7ed6-b807-4eb2-8d31-c55075b1cc4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:03.630609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fb7ed6-b807-4eb2-8d31-c55075b1cc4f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.634329 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85510df-3a45-4136-928d-a6dafff5f9c1 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.637275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.638081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.638403 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.638780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.639843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.642091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.642661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5a65b4-6590-4217-b394-639c28980c20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.643737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5a65b4-6590-4217-b394-639c28980c20 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.644238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5a65b4-6590-4217-b394-639c28980c20 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.644721 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5a65b4-6590-4217-b394-639c28980c20 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.645877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5a65b4-6590-4217-b394-639c28980c20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.649602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5a65b4-6590-4217-b394-639c28980c20 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.666079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1275/2530] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:03 2022] POST /identity/v3/users => generated 291 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:03.667097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.668915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:03.705601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1256/2531] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:03 2022] POST /identity/v3/projects => generated 387 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:03.706711 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5a65b4-6590-4217-b394-639c28980c20 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.752707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:03.762408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6366e2f0-cb24-4fd6-bad1-08b34837f9ea admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.792710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:03.801293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5a65b4-6590-4217-b394-639c28980c20 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.808132 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf53294-2ee5-4a7b-a3fb-cadd9e761705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.808801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf53294-2ee5-4a7b-a3fb-cadd9e761705 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.809013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf53294-2ee5-4a7b-a3fb-cadd9e761705 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.809287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf53294-2ee5-4a7b-a3fb-cadd9e761705 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.809939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf53294-2ee5-4a7b-a3fb-cadd9e761705 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.813080 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf53294-2ee5-4a7b-a3fb-cadd9e761705 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.824933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1257/2532] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:03 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:03.825980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf53294-2ee5-4a7b-a3fb-cadd9e761705 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.838950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf53294-2ee5-4a7b-a3fb-cadd9e761705 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.847951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f79462-8c0f-4da0-80c6-c5eb67b02274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.848578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f79462-8c0f-4da0-80c6-c5eb67b02274 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.848853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f79462-8c0f-4da0-80c6-c5eb67b02274 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.849125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f79462-8c0f-4da0-80c6-c5eb67b02274 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.849763 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f79462-8c0f-4da0-80c6-c5eb67b02274 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.851818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f79462-8c0f-4da0-80c6-c5eb67b02274 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.857000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bd654-e650-4e49-b8c0-ddb9c913749e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.858199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bd654-e650-4e49-b8c0-ddb9c913749e service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.858862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bd654-e650-4e49-b8c0-ddb9c913749e service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.859339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bd654-e650-4e49-b8c0-ddb9c913749e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.876752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7bd654-e650-4e49-b8c0-ddb9c913749e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.879520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7bd654-e650-4e49-b8c0-ddb9c913749e service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.884678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1258/2533] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:03 2022] POST /identity/v3/users => generated 300 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:03.886058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f79462-8c0f-4da0-80c6-c5eb67b02274 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.891676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1276/2534] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:03.892410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bd654-e650-4e49-b8c0-ddb9c913749e service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.903377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f79462-8c0f-4da0-80c6-c5eb67b02274 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.904476 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bd654-e650-4e49-b8c0-ddb9c913749e service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.908744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa2ba7-e368-420f-98f0-f3024b6f8f8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.909656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa2ba7-e368-420f-98f0-f3024b6f8f8a admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.909976 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa2ba7-e368-420f-98f0-f3024b6f8f8a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.910207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fa2ba7-e368-420f-98f0-f3024b6f8f8a admin admin] PATH_INFO: `/v3/projects/0aa77f02f2894088b2d7de10637f67dd/users/bc15281fb3994fb1942fc9eb5f9e4db4/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.910588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558d1c5e-d641-4eb3-b632-f0257614f1d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:03.911962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558d1c5e-d641-4eb3-b632-f0257614f1d6 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:03.912491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558d1c5e-d641-4eb3-b632-f0257614f1d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:03.912808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558d1c5e-d641-4eb3-b632-f0257614f1d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:03.913731 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558d1c5e-d641-4eb3-b632-f0257614f1d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.917484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558d1c5e-d641-4eb3-b632-f0257614f1d6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.928324 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fa2ba7-e368-420f-98f0-f3024b6f8f8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0aa77f02f2894088b2d7de10637f67dd, user_id=bc15281fb3994fb1942fc9eb5f9e4db4, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:03.931588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1277/2535] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:03 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:03.931926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fa2ba7-e368-420f-98f0-f3024b6f8f8a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:03.935687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558d1c5e-d641-4eb3-b632-f0257614f1d6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:03.956183 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558d1c5e-d641-4eb3-b632-f0257614f1d6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:03.956433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1259/2536] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:03 2022] PUT /identity/v3/projects/0aa77f02f2894088b2d7de10637f67dd/users/bc15281fb3994fb1942fc9eb5f9e4db4/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:03.962284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa2ba7-e368-420f-98f0-f3024b6f8f8a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.067829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ec302a-1272-4c6b-94a3-49b4125b14e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.068524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ec302a-1272-4c6b-94a3-49b4125b14e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.068907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ec302a-1272-4c6b-94a3-49b4125b14e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.069198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ec302a-1272-4c6b-94a3-49b4125b14e6 admin admin] PATH_INFO: `/v3/projects/52756e1a03574c42bad50c700848e028/users/42be1b62fee245b190e1f58ecec9bae3/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.076590 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fa2ba7-e368-420f-98f0-f3024b6f8f8a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.085413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f987f-b547-4a66-be3a-0991ca2b68fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.086664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f987f-b547-4a66-be3a-0991ca2b68fe admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.087188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f987f-b547-4a66-be3a-0991ca2b68fe admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.087384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ec302a-1272-4c6b-94a3-49b4125b14e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52756e1a03574c42bad50c700848e028, user_id=42be1b62fee245b190e1f58ecec9bae3, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.087710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f987f-b547-4a66-be3a-0991ca2b68fe admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.088653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622f987f-b547-4a66-be3a-0991ca2b68fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.090068 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ec302a-1272-4c6b-94a3-49b4125b14e6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.090674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622f987f-b547-4a66-be3a-0991ca2b68fe admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.118550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1278/2537] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:03 2022] PUT /identity/v3/projects/52756e1a03574c42bad50c700848e028/users/42be1b62fee245b190e1f58ecec9bae3/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:04.118820 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1260/2538] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:03 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:48:04.122896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ec302a-1272-4c6b-94a3-49b4125b14e6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.124648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f987f-b547-4a66-be3a-0991ca2b68fe admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.138029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f987f-b547-4a66-be3a-0991ca2b68fe admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.226023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f82fd6-da92-4c35-a06c-d757af7e5e5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.228054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f82fd6-da92-4c35-a06c-d757af7e5e5b admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.228054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f82fd6-da92-4c35-a06c-d757af7e5e5b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.228054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f82fd6-da92-4c35-a06c-d757af7e5e5b admin admin] PATH_INFO: `/v3/projects/0aa77f02f2894088b2d7de10637f67dd/users/bc15281fb3994fb1942fc9eb5f9e4db4/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.231482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f82fd6-da92-4c35-a06c-d757af7e5e5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0aa77f02f2894088b2d7de10637f67dd, user_id=bc15281fb3994fb1942fc9eb5f9e4db4, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.234951 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f82fd6-da92-4c35-a06c-d757af7e5e5b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.241802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ec302a-1272-4c6b-94a3-49b4125b14e6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.248885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1261/2539] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:04 2022] PUT /identity/v3/projects/0aa77f02f2894088b2d7de10637f67dd/users/bc15281fb3994fb1942fc9eb5f9e4db4/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:04.259653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007e21f4-7bab-47ca-a501-f312938a8abd None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.259949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007e21f4-7bab-47ca-a501-f312938a8abd None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.260107 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007e21f4-7bab-47ca-a501-f312938a8abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.284666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:04.310533 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-007e21f4-7bab-47ca-a501-f312938a8abd None None] MFA Rules not processed for user `bc15281fb3994fb1942fc9eb5f9e4db4`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:04.313700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-007e21f4-7bab-47ca-a501-f312938a8abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.352314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.353668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.354384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.355263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.359622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.363842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.367936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:04.383163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1279/2540] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:04 2022] GET /identity/v3/roles => generated 1420 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:04.384578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.388906 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.409024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b533a531-2b9c-434f-9b51-fa1bbaa86c76 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.414908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.415779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.418190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.418190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] PATH_INFO: `/v3/projects/91d84016860e45abb691c5163177775d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.420103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91d84016860e45abb691c5163177775d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.422413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.431980 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1262/2541] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:04 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:04.436053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007e21f4-7bab-47ca-a501-f312938a8abd None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.449991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007e21f4-7bab-47ca-a501-f312938a8abd None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.458137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.459438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.460013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.460570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] PATH_INFO: `/v3/projects/52756e1a03574c42bad50c700848e028/users/42be1b62fee245b190e1f58ecec9bae3/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.464349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52756e1a03574c42bad50c700848e028, user_id=42be1b62fee245b190e1f58ecec9bae3, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.470079 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.486780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1263/2542] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:04 2022] PUT /identity/v3/projects/52756e1a03574c42bad50c700848e028/users/42be1b62fee245b190e1f58ecec9bae3/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:04.487771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.491516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.491861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91d84016860e45abb691c5163177775d', 'request_id': 'req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:04.503915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91d84016860e45abb691c5163177775d', 'request_id': 'req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:04.512448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91d84016860e45abb691c5163177775d', 'request_id': 'req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:04.528833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] The token cache is being invalidate because project 91d84016860e45abb691c5163177775d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:04.529129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:04.529848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-705f3ef5-6694-4e1f-a684-2bb3d5b23c49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:04.532816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1280/2543] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:04 2022] DELETE /identity/v3/projects/91d84016860e45abb691c5163177775d => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:04.535529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23e309d-7367-449f-99b1-c2707b4a13e4 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.536058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23e309d-7367-449f-99b1-c2707b4a13e4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.536752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23e309d-7367-449f-99b1-c2707b4a13e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.578555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d23e309d-7367-449f-99b1-c2707b4a13e4 None None] MFA Rules not processed for user `42be1b62fee245b190e1f58ecec9bae3`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:04.581320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d23e309d-7367-449f-99b1-c2707b4a13e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.588278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0db24c-55e5-4d26-906e-89a15edf67d1 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.592634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.593604 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.594047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.594333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.596353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.700743 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.702648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.706463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1281/2544] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:04 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:04.707710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0213050-b720-41ab-9f1b-d4b0ac56509e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.708002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0213050-b720-41ab-9f1b-d4b0ac56509e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.708276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0213050-b720-41ab-9f1b-d4b0ac56509e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.723547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1264/2545] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:04 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:04.725076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.727305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.740564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f0213050-b720-41ab-9f1b-d4b0ac56509e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:04.742502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f0213050-b720-41ab-9f1b-d4b0ac56509e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.746116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20b1126-cefb-4ece-875f-c0cf21109dbb service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.752259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5219ae2-3732-4185-bdf8-deff5f0ff465 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.753216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5219ae2-3732-4185-bdf8-deff5f0ff465 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.753698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5219ae2-3732-4185-bdf8-deff5f0ff465 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.754151 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5219ae2-3732-4185-bdf8-deff5f0ff465 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.776810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5219ae2-3732-4185-bdf8-deff5f0ff465 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.779500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5219ae2-3732-4185-bdf8-deff5f0ff465 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.788989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:04.807041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1265/2546] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:04 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:04.875784 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1282/2547] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:04.891558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29757295-e96f-468b-ad1c-1782b52f3b20 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.892280 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29757295-e96f-468b-ad1c-1782b52f3b20 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.892609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29757295-e96f-468b-ad1c-1782b52f3b20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.912221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0213050-b720-41ab-9f1b-d4b0ac56509e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.914767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f0213050-b720-41ab-9f1b-d4b0ac56509e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.934386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-29757295-e96f-468b-ad1c-1782b52f3b20 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:04.935932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-29757295-e96f-468b-ad1c-1782b52f3b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:04.939018 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0213050-b720-41ab-9f1b-d4b0ac56509e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.948896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1266/2548] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:04.949724 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.950891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.951405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.951911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] PATH_INFO: `/v3/users/ae5fbe31ee71437cbf18bfcfe451d19e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.954806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae5fbe31ee71437cbf18bfcfe451d19e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.955348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29757295-e96f-468b-ad1c-1782b52f3b20 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:04.958886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:04.975635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29757295-e96f-468b-ad1c-1782b52f3b20 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:04.984616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:04.986062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:04.986340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:04.986624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:04.987315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:04.987624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:04.989293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:05.007200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1267/2549] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:05.014573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:05.034437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecfca7a-ad1f-46a6-b84d-644519462112 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:05.041605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:05.042267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.042565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.042798 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.043687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:05.045926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:05.062165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae5fbe31ee71437cbf18bfcfe451d19e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:05.073965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae5fbe31ee71437cbf18bfcfe451d19e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:05.075130 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae5fbe31ee71437cbf18bfcfe451d19e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:05.076064 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae5fbe31ee71437cbf18bfcfe451d19e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:05.085190 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1268/2550] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:05 2022] POST /identity/v3/projects => generated 375 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:05.088145 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3607df08-1b2b-480e-a2c6-54c2f9166e03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae5fbe31ee71437cbf18bfcfe451d19e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:05.091517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:05.094502 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:05.117504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1283/2551] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:04 2022] DELETE /identity/v3/users/ae5fbe31ee71437cbf18bfcfe451d19e => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:05.122751 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:05.123268 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd9939-5afc-42df-9d7e-b78bc82488b3 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.123649 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd9939-5afc-42df-9d7e-b78bc82488b3 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.124238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd9939-5afc-42df-9d7e-b78bc82488b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.148415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-33fd9939-5afc-42df-9d7e-b78bc82488b3 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:05.149636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-33fd9939-5afc-42df-9d7e-b78bc82488b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:05.223668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cbdaae-273a-4731-ae21-a9aefb0beebe admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:05.265750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1284/2552] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:05.282414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fd9939-5afc-42df-9d7e-b78bc82488b3 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:05.285525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-33fd9939-5afc-42df-9d7e-b78bc82488b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:05.320367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e4deef-7494-4fda-ae4c-52fdc10938e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:05.321111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e4deef-7494-4fda-ae4c-52fdc10938e9 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.321455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e4deef-7494-4fda-ae4c-52fdc10938e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.321800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e4deef-7494-4fda-ae4c-52fdc10938e9 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.322669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e4deef-7494-4fda-ae4c-52fdc10938e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:05.324727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e4deef-7494-4fda-ae4c-52fdc10938e9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:05.352423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1269/2553] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:05 2022] POST /identity/v3/users => generated 288 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:05.359053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e4deef-7494-4fda-ae4c-52fdc10938e9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:05.369582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:05.382856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e4deef-7494-4fda-ae4c-52fdc10938e9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:05.390467 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aed53e-471b-4a17-97cb-c557c655b1d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:05.391027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fd9939-5afc-42df-9d7e-b78bc82488b3 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:05.391207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aed53e-471b-4a17-97cb-c557c655b1d7 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.391578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aed53e-471b-4a17-97cb-c557c655b1d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.391999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aed53e-471b-4a17-97cb-c557c655b1d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.392956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aed53e-471b-4a17-97cb-c557c655b1d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:05.395450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aed53e-471b-4a17-97cb-c557c655b1d7 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:05.395662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca04704a-50dd-41f6-af2e-fa36e4257f45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:05.396371 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04704a-50dd-41f6-af2e-fa36e4257f45 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.396721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04704a-50dd-41f6-af2e-fa36e4257f45 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.397158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04704a-50dd-41f6-af2e-fa36e4257f45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.408670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca04704a-50dd-41f6-af2e-fa36e4257f45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:05.409500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1270/2554] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:05 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:05.410223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca04704a-50dd-41f6-af2e-fa36e4257f45 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:05.420861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aed53e-471b-4a17-97cb-c557c655b1d7 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:05.428251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1285/2555] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:05.443910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aed53e-471b-4a17-97cb-c557c655b1d7 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:05.467903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605c763-2071-4a90-a6db-8946d5471afe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:05.469103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605c763-2071-4a90-a6db-8946d5471afe admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.472024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605c763-2071-4a90-a6db-8946d5471afe admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.472024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605c763-2071-4a90-a6db-8946d5471afe admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.472024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605c763-2071-4a90-a6db-8946d5471afe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:05.473861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605c763-2071-4a90-a6db-8946d5471afe admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:05.500960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1271/2556] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:05 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:05.502419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca04704a-50dd-41f6-af2e-fa36e4257f45 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:05.519775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca04704a-50dd-41f6-af2e-fa36e4257f45 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:05.530011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:05.531186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.531877 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.532369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] PATH_INFO: `/v3/projects/4462ca02650f47d4bcd2a74789260a61/users/aea0676377ae4d95991a7cf766bd6cce/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.565464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4462ca02650f47d4bcd2a74789260a61, user_id=aea0676377ae4d95991a7cf766bd6cce, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:05.570349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:05.596764 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1286/2557] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:05 2022] PUT /identity/v3/projects/4462ca02650f47d4bcd2a74789260a61/users/aea0676377ae4d95991a7cf766bd6cce/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:05.618231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116f440f-e90f-48ff-995b-d906844401cc None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.618675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116f440f-e90f-48ff-995b-d906844401cc None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.619092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116f440f-e90f-48ff-995b-d906844401cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.669966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-116f440f-e90f-48ff-995b-d906844401cc None None] MFA Rules not processed for user `aea0676377ae4d95991a7cf766bd6cce`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:05.671650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-116f440f-e90f-48ff-995b-d906844401cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:05.820323 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1272/2558] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:05 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:05.835883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:05.837620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:05.931422 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116f440f-e90f-48ff-995b-d906844401cc None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:05.934375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-116f440f-e90f-48ff-995b-d906844401cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:05.949635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d063ac-acf3-430f-b254-b878714a0cb4 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:05.956098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:05.959553 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d4b8c0-224b-4f36-9069-41694ea2a1ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:05.960728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d4b8c0-224b-4f36-9069-41694ea2a1ab service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:05.961251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d4b8c0-224b-4f36-9069-41694ea2a1ab service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:05.963017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d4b8c0-224b-4f36-9069-41694ea2a1ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:05.975104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d4b8c0-224b-4f36-9069-41694ea2a1ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:05.977155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d4b8c0-224b-4f36-9069-41694ea2a1ab service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:05.998604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1287/2559] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:05 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:06.082431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116f440f-e90f-48ff-995b-d906844401cc None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:06.094880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:06.096251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:06.097292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:06.097971 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] PATH_INFO: `/v3/projects/ac895a47589d48b99cb6f3964579c121` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:06.100001 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac895a47589d48b99cb6f3964579c121)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:06.102292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:06.191825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac895a47589d48b99cb6f3964579c121', 'request_id': 'req-f402e7bc-4543-4cff-9195-5f4bba5cd135'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.200801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac895a47589d48b99cb6f3964579c121', 'request_id': 'req-f402e7bc-4543-4cff-9195-5f4bba5cd135'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.214494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac895a47589d48b99cb6f3964579c121', 'request_id': 'req-f402e7bc-4543-4cff-9195-5f4bba5cd135'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.236543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] The token cache is being invalidate because project ac895a47589d48b99cb6f3964579c121 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:06.237025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.238370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.244123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1273/2560] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:05 2022] DELETE /identity/v3/projects/ac895a47589d48b99cb6f3964579c121 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:06.245623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d4b8c0-224b-4f36-9069-41694ea2a1ab service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:06.271927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d4b8c0-224b-4f36-9069-41694ea2a1ab service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:06.392047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:06.392942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:06.393381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:06.393805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] PATH_INFO: `/v3/users/42be1b62fee245b190e1f58ecec9bae3` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:06.395559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=42be1b62fee245b190e1f58ecec9bae3)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:06.398040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:06.503852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42be1b62fee245b190e1f58ecec9bae3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.518813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42be1b62fee245b190e1f58ecec9bae3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.521436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42be1b62fee245b190e1f58ecec9bae3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.523135 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42be1b62fee245b190e1f58ecec9bae3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.540468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42be1b62fee245b190e1f58ecec9bae3'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:06.545623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:06.579031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1288/2561] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:06 2022] DELETE /identity/v3/users/42be1b62fee245b190e1f58ecec9bae3 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:07.020547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.027303 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:07.047046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.050856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f402e7bc-4543-4cff-9195-5f4bba5cd135 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.056796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:07.159858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.160870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.161261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.161745 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] PATH_INFO: `/v3/users/2498e216739545418885a7c6e03a773a` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.163903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2498e216739545418885a7c6e03a773a)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.166889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:07.177717 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2304acb-9d62-45c0-a34c-01019c0467f8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.187454 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.188765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.189269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.189713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] PATH_INFO: `/v3/projects/52756e1a03574c42bad50c700848e028` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.191603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=52756e1a03574c42bad50c700848e028)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.195612 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:07.251449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:07.255634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2498e216739545418885a7c6e03a773a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.256999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2498e216739545418885a7c6e03a773a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.269014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52756e1a03574c42bad50c700848e028', 'request_id': 'req-50fe8e26-5992-469a-8f97-3de10a015d9c'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.271411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2498e216739545418885a7c6e03a773a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.276331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52756e1a03574c42bad50c700848e028', 'request_id': 'req-50fe8e26-5992-469a-8f97-3de10a015d9c'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.283296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2498e216739545418885a7c6e03a773a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.284675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52756e1a03574c42bad50c700848e028', 'request_id': 'req-50fe8e26-5992-469a-8f97-3de10a015d9c'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.307867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-512a63d7-8789-4b00-8c5e-63fd9a937c26 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2498e216739545418885a7c6e03a773a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.317030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] The token cache is being invalidate because project 52756e1a03574c42bad50c700848e028 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:07.317404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.318318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-50fe8e26-5992-469a-8f97-3de10a015d9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:07.319799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1274/2562] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:07 2022] DELETE /identity/v3/users/2498e216739545418885a7c6e03a773a => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:07.320405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1289/2563] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:07 2022] DELETE /identity/v3/projects/52756e1a03574c42bad50c700848e028 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:07.326849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4275316-aeff-436c-a370-fb04e737d56d None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.327109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4275316-aeff-436c-a370-fb04e737d56d None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.327974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4275316-aeff-436c-a370-fb04e737d56d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.330038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc96fb-ee30-46b0-b0d5-80bcd9158f0c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.330801 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc96fb-ee30-46b0-b0d5-80bcd9158f0c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.331515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc96fb-ee30-46b0-b0d5-80bcd9158f0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.361140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e4275316-aeff-436c-a370-fb04e737d56d None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:07.362425 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e4275316-aeff-436c-a370-fb04e737d56d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:07.363675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e9dc96fb-ee30-46b0-b0d5-80bcd9158f0c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:07.365431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e9dc96fb-ee30-46b0-b0d5-80bcd9158f0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:07.435727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1290/2564] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:07.448884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dc96fb-ee30-46b0-b0d5-80bcd9158f0c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.451627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e9dc96fb-ee30-46b0-b0d5-80bcd9158f0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:07.453640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1275/2565] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:07.464473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d07e1-b405-49d0-be74-da86998db002 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.466836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d07e1-b405-49d0-be74-da86998db002 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.467827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d07e1-b405-49d0-be74-da86998db002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.478802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:07.498162 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-b56d07e1-b405-49d0-be74-da86998db002 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:07.500287 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b56d07e1-b405-49d0-be74-da86998db002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:07.513629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1276/2566] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:07.519411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56d07e1-b405-49d0-be74-da86998db002 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.531243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56d07e1-b405-49d0-be74-da86998db002 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.536561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.537423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.537605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.537949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.538782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:07.539140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.540944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:07.558169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1277/2567] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:07.570725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.572872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dc96fb-ee30-46b0-b0d5-80bcd9158f0c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.578920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5599dd16-4245-49d4-b701-8a00cf4ca64b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.579171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5346214-f762-4ca7-b055-4bae2ca6cda8 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.579675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5599dd16-4245-49d4-b701-8a00cf4ca64b service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.580026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5599dd16-4245-49d4-b701-8a00cf4ca64b service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.580375 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5599dd16-4245-49d4-b701-8a00cf4ca64b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.584772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43c6ece-112c-4cd3-b15e-b4c203c598a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.585512 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43c6ece-112c-4cd3-b15e-b4c203c598a4 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.585845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43c6ece-112c-4cd3-b15e-b4c203c598a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.586291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43c6ece-112c-4cd3-b15e-b4c203c598a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.587156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43c6ece-112c-4cd3-b15e-b4c203c598a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.589335 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43c6ece-112c-4cd3-b15e-b4c203c598a4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:07.595117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5599dd16-4245-49d4-b701-8a00cf4ca64b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.598460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5599dd16-4245-49d4-b701-8a00cf4ca64b service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:07.615250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1291/2568] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:07.633490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1278/2569] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:07 2022] POST /identity/v3/projects => generated 379 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:07.642167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5599dd16-4245-49d4-b701-8a00cf4ca64b service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.657726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5599dd16-4245-49d4-b701-8a00cf4ca64b service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.745317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca860642-828c-4af2-8f5b-b09ec3f209ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.746207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca860642-828c-4af2-8f5b-b09ec3f209ef admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.747043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca860642-828c-4af2-8f5b-b09ec3f209ef admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.747043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca860642-828c-4af2-8f5b-b09ec3f209ef admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.747934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca860642-828c-4af2-8f5b-b09ec3f209ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.751491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca860642-828c-4af2-8f5b-b09ec3f209ef admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:07.780181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1292/2570] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:07 2022] POST /identity/v3/users => generated 290 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:07.796977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43c6ece-112c-4cd3-b15e-b4c203c598a4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.835226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43c6ece-112c-4cd3-b15e-b4c203c598a4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.849701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f80fd4-dd84-48df-a73f-e573a1aa336c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.849701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f80fd4-dd84-48df-a73f-e573a1aa336c admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.849701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f80fd4-dd84-48df-a73f-e573a1aa336c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.850319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f80fd4-dd84-48df-a73f-e573a1aa336c admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.851520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f80fd4-dd84-48df-a73f-e573a1aa336c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.855099 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f80fd4-dd84-48df-a73f-e573a1aa336c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:07.872380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1279/2571] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:07.878791 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca860642-828c-4af2-8f5b-b09ec3f209ef admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.896499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca860642-828c-4af2-8f5b-b09ec3f209ef admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.902478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.903382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.903754 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.904120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.904905 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.907241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:07.935101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1293/2572] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:07.941842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f80fd4-dd84-48df-a73f-e573a1aa336c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.957932 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f80fd4-dd84-48df-a73f-e573a1aa336c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.963138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ae452f-9832-474a-9644-3c834f2c8ebf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:07.963797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ae452f-9832-474a-9644-3c834f2c8ebf admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:07.964072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ae452f-9832-474a-9644-3c834f2c8ebf admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:07.964353 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ae452f-9832-474a-9644-3c834f2c8ebf admin admin] PATH_INFO: `/v3/projects/fabbc00fc4634fd9a23d879e64518aed/users/114db9ee5a2341afb6e442ff447c0150/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:07.972963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:07.977410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:07.995285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ae452f-9832-474a-9644-3c834f2c8ebf admin admin] RBAC: Authorizing `identity:create_grant(project_id=fabbc00fc4634fd9a23d879e64518aed, user_id=114db9ee5a2341afb6e442ff447c0150, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:07.996541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cd67b3-f7a2-400c-bba9-1f884797c7e2 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:07.999885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ae452f-9832-474a-9644-3c834f2c8ebf admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.003086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.004174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.004464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.004736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] PATH_INFO: `/v3/projects/d21f41e8a9b6429a8871af251eb9df68` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.006779 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d21f41e8a9b6429a8871af251eb9df68)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.007998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.023458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1280/2573] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:07 2022] PUT /identity/v3/projects/fabbc00fc4634fd9a23d879e64518aed/users/114db9ee5a2341afb6e442ff447c0150/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:08.039516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d179ae-1e0f-4b00-8d12-821e325be084 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.040436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d179ae-1e0f-4b00-8d12-821e325be084 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.041285 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d179ae-1e0f-4b00-8d12-821e325be084 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.070401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd21f41e8a9b6429a8871af251eb9df68', 'request_id': 'req-451568a9-565d-45c4-85ae-3d7766dad65b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.079168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd21f41e8a9b6429a8871af251eb9df68', 'request_id': 'req-451568a9-565d-45c4-85ae-3d7766dad65b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.083861 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-35d179ae-1e0f-4b00-8d12-821e325be084 None None] MFA Rules not processed for user `114db9ee5a2341afb6e442ff447c0150`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:08.084998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-35d179ae-1e0f-4b00-8d12-821e325be084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:08.088566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd21f41e8a9b6429a8871af251eb9df68', 'request_id': 'req-451568a9-565d-45c4-85ae-3d7766dad65b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.113302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] The token cache is being invalidate because project d21f41e8a9b6429a8871af251eb9df68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:08.113720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.114562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.116727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1294/2574] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:07 2022] DELETE /identity/v3/projects/d21f41e8a9b6429a8871af251eb9df68 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:08.121028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:08.122486 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:08.131659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:08.157473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:08.194081 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1281/2575] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:08 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:08.196115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d179ae-1e0f-4b00-8d12-821e325be084 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:08.198347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-35d179ae-1e0f-4b00-8d12-821e325be084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:08.237420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451568a9-565d-45c4-85ae-3d7766dad65b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:08.245356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.246518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.247073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.247578 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] PATH_INFO: `/v3/users/bc15281fb3994fb1942fc9eb5f9e4db4` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.250193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc15281fb3994fb1942fc9eb5f9e4db4)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.252716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.303699 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d179ae-1e0f-4b00-8d12-821e325be084 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:08.311220 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.312481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.312797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.313086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.313726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.316686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.323788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc15281fb3994fb1942fc9eb5f9e4db4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.333030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc15281fb3994fb1942fc9eb5f9e4db4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.333889 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc15281fb3994fb1942fc9eb5f9e4db4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.334575 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc15281fb3994fb1942fc9eb5f9e4db4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.345067 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc15281fb3994fb1942fc9eb5f9e4db4'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:08.357385 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1282/2576] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:08 2022] POST /identity/v3/projects => generated 381 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:08.362516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:08.364576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:08.373485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1295/2577] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:08 2022] DELETE /identity/v3/users/bc15281fb3994fb1942fc9eb5f9e4db4 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:08.457713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb1cce-9f22-49b6-83cd-7f8ffd9d7174 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:08.538888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c58c1-60d0-43ef-9e52-69267bc55f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.539519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c58c1-60d0-43ef-9e52-69267bc55f9e admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.539797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c58c1-60d0-43ef-9e52-69267bc55f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.540067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c58c1-60d0-43ef-9e52-69267bc55f9e admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.540713 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17c58c1-60d0-43ef-9e52-69267bc55f9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.542726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17c58c1-60d0-43ef-9e52-69267bc55f9e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.559873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:08.576656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1283/2578] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:08 2022] POST /identity/v3/users => generated 297 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:08.586818 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:08.611467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8098a5b8-442f-4e45-98b0-6e0fecf734f8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:08.617921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc20815b-7059-452d-903b-a9e8104a8924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.618676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20815b-7059-452d-903b-a9e8104a8924 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.619048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20815b-7059-452d-903b-a9e8104a8924 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.619320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20815b-7059-452d-903b-a9e8104a8924 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.620208 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc20815b-7059-452d-903b-a9e8104a8924 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.621977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc20815b-7059-452d-903b-a9e8104a8924 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.637888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1296/2579] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:08.639492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c58c1-60d0-43ef-9e52-69267bc55f9e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:08.651898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c58c1-60d0-43ef-9e52-69267bc55f9e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:08.657451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.658621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.658621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.659030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] PATH_INFO: `/v3/projects/4e3663adceb446a4847b125691dbd0a7/users/5edff732a0bf4d11be444ae666801854/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.684896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e3663adceb446a4847b125691dbd0a7, user_id=5edff732a0bf4d11be444ae666801854, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.691685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.720312 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1284/2580] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:08 2022] PUT /identity/v3/projects/4e3663adceb446a4847b125691dbd0a7/users/5edff732a0bf4d11be444ae666801854/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:08.724715 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc20815b-7059-452d-903b-a9e8104a8924 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:08.741218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc20815b-7059-452d-903b-a9e8104a8924 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:08.827862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:08.831482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:08.911402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0ea032-3439-4ddb-81c6-7e46a771c52e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.912451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0ea032-3439-4ddb-81c6-7e46a771c52e admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.912944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0ea032-3439-4ddb-81c6-7e46a771c52e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.913617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0ea032-3439-4ddb-81c6-7e46a771c52e admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.914520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0ea032-3439-4ddb-81c6-7e46a771c52e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.917308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0ea032-3439-4ddb-81c6-7e46a771c52e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.937058 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1297/2581] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:08.942699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0ea032-3439-4ddb-81c6-7e46a771c52e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:08.955752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e55fbb-0b9b-4b49-9885-f8e6739c54ed admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:08.961777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.962446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.962783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.963046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] PATH_INFO: `/v3/projects/0aa77f02f2894088b2d7de10637f67dd` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.964468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0aa77f02f2894088b2d7de10637f67dd)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.965383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0ea032-3439-4ddb-81c6-7e46a771c52e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:08.966439 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:08.983107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d5e698-51fe-4aa5-9fa6-3ca6e07aed7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:08.984747 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5e698-51fe-4aa5-9fa6-3ca6e07aed7b admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:08.985118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5e698-51fe-4aa5-9fa6-3ca6e07aed7b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:08.985571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5e698-51fe-4aa5-9fa6-3ca6e07aed7b admin admin] PATH_INFO: `/v3/projects/4e3663adceb446a4847b125691dbd0a7/users/5edff732a0bf4d11be444ae666801854/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:08.989174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d5e698-51fe-4aa5-9fa6-3ca6e07aed7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e3663adceb446a4847b125691dbd0a7, user_id=5edff732a0bf4d11be444ae666801854, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:08.997668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d5e698-51fe-4aa5-9fa6-3ca6e07aed7b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.024561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1298/2582] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:08 2022] PUT /identity/v3/projects/4e3663adceb446a4847b125691dbd0a7/users/5edff732a0bf4d11be444ae666801854/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:09.037449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e052c33-ec0f-4f51-97ba-1bf647144644 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.037821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e052c33-ec0f-4f51-97ba-1bf647144644 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.038176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e052c33-ec0f-4f51-97ba-1bf647144644 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.067173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:09.072336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0aa77f02f2894088b2d7de10637f67dd', 'request_id': 'req-eb44b15a-3b14-4624-9d82-c8b6aa259e38'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:09.081341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2e052c33-ec0f-4f51-97ba-1bf647144644 None None] MFA Rules not processed for user `5edff732a0bf4d11be444ae666801854`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:09.082212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0aa77f02f2894088b2d7de10637f67dd', 'request_id': 'req-eb44b15a-3b14-4624-9d82-c8b6aa259e38'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:09.083317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2e052c33-ec0f-4f51-97ba-1bf647144644 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:09.091770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0aa77f02f2894088b2d7de10637f67dd', 'request_id': 'req-eb44b15a-3b14-4624-9d82-c8b6aa259e38'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:09.107939 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] The token cache is being invalidate because project 0aa77f02f2894088b2d7de10637f67dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:09.109163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:09.110046 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-eb44b15a-3b14-4624-9d82-c8b6aa259e38 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:09.111716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1285/2583] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:08 2022] DELETE /identity/v3/projects/0aa77f02f2894088b2d7de10637f67dd => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:09.116599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783daf10-b61b-4be4-9100-7151ab1ad37c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.116869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783daf10-b61b-4be4-9100-7151ab1ad37c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.117147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783daf10-b61b-4be4-9100-7151ab1ad37c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.142387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-783daf10-b61b-4be4-9100-7151ab1ad37c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:09.143610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-783daf10-b61b-4be4-9100-7151ab1ad37c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:09.203390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1299/2584] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:09 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:09.218960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e052c33-ec0f-4f51-97ba-1bf647144644 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.221725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2e052c33-ec0f-4f51-97ba-1bf647144644 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:09.243468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1286/2585] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:09.254149 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272fc7e-b06f-46a4-a0d2-1f61535e0884 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.255223 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272fc7e-b06f-46a4-a0d2-1f61535e0884 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.255804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272fc7e-b06f-46a4-a0d2-1f61535e0884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.267432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:09.288639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5272fc7e-b06f-46a4-a0d2-1f61535e0884 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:09.290476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5272fc7e-b06f-46a4-a0d2-1f61535e0884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:09.300260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1287/2586] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:09.305446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5272fc7e-b06f-46a4-a0d2-1f61535e0884 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.310047 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e052c33-ec0f-4f51-97ba-1bf647144644 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.314607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e477e2fc-8eda-4289-8885-a7d25e7d7ac9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.315296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e477e2fc-8eda-4289-8885-a7d25e7d7ac9 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.315572 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e477e2fc-8eda-4289-8885-a7d25e7d7ac9 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.315838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e477e2fc-8eda-4289-8885-a7d25e7d7ac9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.319288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5272fc7e-b06f-46a4-a0d2-1f61535e0884 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.326383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.327433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.327891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.328319 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.329426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:09.329922 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.332445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e477e2fc-8eda-4289-8885-a7d25e7d7ac9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.332981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.335265 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e477e2fc-8eda-4289-8885-a7d25e7d7ac9 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.354127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1288/2587] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:48:09.357016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1300/2588] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:09 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:09.359128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.370491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c3cfda-fa01-4ba2-b3c8-460ef44939d4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.375686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83e37e-7434-4a98-bd09-5f738dfe2eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.376394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83e37e-7434-4a98-bd09-5f738dfe2eca admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.376679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83e37e-7434-4a98-bd09-5f738dfe2eca admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.376957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83e37e-7434-4a98-bd09-5f738dfe2eca admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.377621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b83e37e-7434-4a98-bd09-5f738dfe2eca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.380281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b83e37e-7434-4a98-bd09-5f738dfe2eca admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.418077 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1289/2589] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:09 2022] POST /identity/v3/projects => generated 369 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:09.425282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e477e2fc-8eda-4289-8885-a7d25e7d7ac9 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.444239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e477e2fc-8eda-4289-8885-a7d25e7d7ac9 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.540302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4baa0801-a3c7-4039-9b30-48dbe0e27b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.541247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baa0801-a3c7-4039-9b30-48dbe0e27b9e admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.541704 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baa0801-a3c7-4039-9b30-48dbe0e27b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.542154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baa0801-a3c7-4039-9b30-48dbe0e27b9e admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.543166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4baa0801-a3c7-4039-9b30-48dbe0e27b9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.546761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4baa0801-a3c7-4039-9b30-48dbe0e27b9e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.577193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1301/2590] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:09 2022] POST /identity/v3/users => generated 285 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:09.581593 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83e37e-7434-4a98-bd09-5f738dfe2eca admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.594502 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83e37e-7434-4a98-bd09-5f738dfe2eca admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.600033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ddde12-e047-4123-9793-a968a97a4a36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.600773 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddde12-e047-4123-9793-a968a97a4a36 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.601127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddde12-e047-4123-9793-a968a97a4a36 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.601402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddde12-e047-4123-9793-a968a97a4a36 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.602061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ddde12-e047-4123-9793-a968a97a4a36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.603737 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ddde12-e047-4123-9793-a968a97a4a36 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.620031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1290/2591] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:09.624450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4baa0801-a3c7-4039-9b30-48dbe0e27b9e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.649410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4baa0801-a3c7-4039-9b30-48dbe0e27b9e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.657340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e95bab-b809-4118-881f-0dea63bac06f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.658269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e95bab-b809-4118-881f-0dea63bac06f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.658862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e95bab-b809-4118-881f-0dea63bac06f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.659413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e95bab-b809-4118-881f-0dea63bac06f admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.660650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e95bab-b809-4118-881f-0dea63bac06f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.663410 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e95bab-b809-4118-881f-0dea63bac06f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.678158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1302/2592] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:48:09.682561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ddde12-e047-4123-9793-a968a97a4a36 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.701041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ddde12-e047-4123-9793-a968a97a4a36 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.708883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cb17f1-9fe6-4105-972e-f2a59fe6e468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.709990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cb17f1-9fe6-4105-972e-f2a59fe6e468 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.710221 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cb17f1-9fe6-4105-972e-f2a59fe6e468 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.710626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cb17f1-9fe6-4105-972e-f2a59fe6e468 admin admin] PATH_INFO: `/v3/projects/2a9607b88fb446ec965f0f560c6a73b5/users/662c210639dd4065ae46b18fd2737e53/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.730212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cb17f1-9fe6-4105-972e-f2a59fe6e468 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a9607b88fb446ec965f0f560c6a73b5, user_id=662c210639dd4065ae46b18fd2737e53, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.734128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cb17f1-9fe6-4105-972e-f2a59fe6e468 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.751941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1291/2593] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:09 2022] PUT /identity/v3/projects/2a9607b88fb446ec965f0f560c6a73b5/users/662c210639dd4065ae46b18fd2737e53/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:09.758361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2340ab7-acff-4b9b-ba39-3b976cf35ef8 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.758673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2340ab7-acff-4b9b-ba39-3b976cf35ef8 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.759561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2340ab7-acff-4b9b-ba39-3b976cf35ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.792169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d2340ab7-acff-4b9b-ba39-3b976cf35ef8 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:09.792169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d2340ab7-acff-4b9b-ba39-3b976cf35ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:09.794205 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cb17f1-9fe6-4105-972e-f2a59fe6e468 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.800101 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:09.884407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1303/2594] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:09.897405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2340ab7-acff-4b9b-ba39-3b976cf35ef8 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.899118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d2340ab7-acff-4b9b-ba39-3b976cf35ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:09.900110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cb17f1-9fe6-4105-972e-f2a59fe6e468 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.908226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5edff732a0bf4d11be444ae666801854', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4e3663adceb446a4847b125691dbd0a7', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.909112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.909621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.910089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.911277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:09.911790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.914902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.942621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1292/2595] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:48:09.944042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:09.962414 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e21a309-c62a-4210-8b48-b34acbcc1991 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.971261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2051f373-13b9-4e37-b755-1d3d5f5687c1 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5edff732a0bf4d11be444ae666801854', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4e3663adceb446a4847b125691dbd0a7', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.972432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2051f373-13b9-4e37-b755-1d3d5f5687c1 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.972948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2051f373-13b9-4e37-b755-1d3d5f5687c1 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.973494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2051f373-13b9-4e37-b755-1d3d5f5687c1 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:09.974559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2051f373-13b9-4e37-b755-1d3d5f5687c1 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:09.978322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2051f373-13b9-4e37-b755-1d3d5f5687c1 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:09.988443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2340ab7-acff-4b9b-ba39-3b976cf35ef8 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:09.992270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6fc408-acd9-45d7-9b82-752229f212fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:09.992872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6fc408-acd9-45d7-9b82-752229f212fa service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:09.993147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6fc408-acd9-45d7-9b82-752229f212fa service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:09.993418 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6fc408-acd9-45d7-9b82-752229f212fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:10.008964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6fc408-acd9-45d7-9b82-752229f212fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:10.010470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6fc408-acd9-45d7-9b82-752229f212fa service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:10.023279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1304/2596] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:10.023600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1293/2597] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:09 2022] POST /identity/v3/projects => generated 355 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:10.027762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6fc408-acd9-45d7-9b82-752229f212fa service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:10.040176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6fc408-acd9-45d7-9b82-752229f212fa service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:10.120998 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5edff732a0bf4d11be444ae666801854', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4e3663adceb446a4847b125691dbd0a7', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:10.121947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:10.122063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:10.122377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:10.123163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:10.123550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:10.125718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:10.155264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1305/2598] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:48:10.221288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2051f373-13b9-4e37-b755-1d3d5f5687c1 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:10.236579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2051f373-13b9-4e37-b755-1d3d5f5687c1 tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:10.245111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5edff732a0bf4d11be444ae666801854', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4e3663adceb446a4847b125691dbd0a7', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:10.245730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:10.246020 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:10.246367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] PATH_INFO: `/v3/projects/4bf71846fb3f46798c94ba2263d62ce2` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:10.247849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Authorizing `identity:delete_project(project_id=4bf71846fb3f46798c94ba2263d62ce2)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:10.250496 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:10.308143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4bf71846fb3f46798c94ba2263d62ce2', 'request_id': 'req-22375546-2b99-4aff-a7f7-ca2d8753185f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.310159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:10.316599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4bf71846fb3f46798c94ba2263d62ce2', 'request_id': 'req-22375546-2b99-4aff-a7f7-ca2d8753185f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.330914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4bf71846fb3f46798c94ba2263d62ce2', 'request_id': 'req-22375546-2b99-4aff-a7f7-ca2d8753185f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.345165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] The token cache is being invalidate because project 4bf71846fb3f46798c94ba2263d62ce2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:10.345498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.346167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-22375546-2b99-4aff-a7f7-ca2d8753185f tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.348171 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1294/2599] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:10 2022] DELETE /identity/v3/projects/4bf71846fb3f46798c94ba2263d62ce2 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:10.626017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:10.627977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:10.703879 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:10.723752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4da31e5-caf0-48cd-94f3-ed90a68a944b tempest-VolumeQuotaClassesTest-1765536549 tempest-VolumeQuotaClassesTest-1765536549-project-admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:10.813373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:10.814221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:10.814709 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:10.815124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] PATH_INFO: `/v3/users/114db9ee5a2341afb6e442ff447c0150` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:10.819419 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] RBAC: Authorizing `identity:delete_user(user_id=114db9ee5a2341afb6e442ff447c0150)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:10.823176 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:10.898579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72cab10-bc26-4988-8d7a-02c73f91db8e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:10.899009 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72cab10-bc26-4988-8d7a-02c73f91db8e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:10.899307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72cab10-bc26-4988-8d7a-02c73f91db8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:10.919200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '114db9ee5a2341afb6e442ff447c0150'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.936327 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '114db9ee5a2341afb6e442ff447c0150'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.937282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '114db9ee5a2341afb6e442ff447c0150'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.937931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '114db9ee5a2341afb6e442ff447c0150'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.938131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c72cab10-bc26-4988-8d7a-02c73f91db8e None None] MFA Rules not processed for user `662c210639dd4065ae46b18fd2737e53`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:10.940557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c72cab10-bc26-4988-8d7a-02c73f91db8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:10.950577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f8b6ed8d-50e7-4ed8-8580-a46b007d7765 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '114db9ee5a2341afb6e442ff447c0150'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:10.998882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1306/2600] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:10 2022] DELETE /identity/v3/users/114db9ee5a2341afb6e442ff447c0150 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:11.003406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b11e53-cdd7-4e0a-844e-e1bf7f241a22 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.003693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b11e53-cdd7-4e0a-844e-e1bf7f241a22 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.003966 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b11e53-cdd7-4e0a-844e-e1bf7f241a22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:11.028977 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-60b11e53-cdd7-4e0a-844e-e1bf7f241a22 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:11.032775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-60b11e53-cdd7-4e0a-844e-e1bf7f241a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:11.070050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1295/2601] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:10 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:11.071028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72cab10-bc26-4988-8d7a-02c73f91db8e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:11.072608 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c72cab10-bc26-4988-8d7a-02c73f91db8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:11.091095 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72cab10-bc26-4988-8d7a-02c73f91db8e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:11.142259 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1307/2602] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:11.159349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b11e53-cdd7-4e0a-844e-e1bf7f241a22 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:11.161528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-60b11e53-cdd7-4e0a-844e-e1bf7f241a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:11.195970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc77af7-e0f5-4016-a3f6-e4fdd9d21855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:11.197075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc77af7-e0f5-4016-a3f6-e4fdd9d21855 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.197672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc77af7-e0f5-4016-a3f6-e4fdd9d21855 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.198011 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc77af7-e0f5-4016-a3f6-e4fdd9d21855 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:11.203093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc77af7-e0f5-4016-a3f6-e4fdd9d21855 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:11.203093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc77af7-e0f5-4016-a3f6-e4fdd9d21855 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:11.259667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1296/2603] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:11 2022] POST /identity/v3/projects => generated 371 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:11.270315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc77af7-e0f5-4016-a3f6-e4fdd9d21855 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:11.337309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:11.363687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:11.373527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc77af7-e0f5-4016-a3f6-e4fdd9d21855 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:11.380427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b11e53-cdd7-4e0a-844e-e1bf7f241a22 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:11.389535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d8c708-158f-4cd9-a8e8-a4fb08618513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:11.390708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8c708-158f-4cd9-a8e8-a4fb08618513 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.391014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8c708-158f-4cd9-a8e8-a4fb08618513 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.391283 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8c708-158f-4cd9-a8e8-a4fb08618513 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:11.391928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d8c708-158f-4cd9-a8e8-a4fb08618513 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:11.393881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d8c708-158f-4cd9-a8e8-a4fb08618513 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:11.427525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1297/2604] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:11 2022] POST /identity/v3/users => generated 292 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:11.429241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d8c708-158f-4cd9-a8e8-a4fb08618513 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:11.442379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d8c708-158f-4cd9-a8e8-a4fb08618513 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:11.449240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a42589d-1def-4b0a-afbe-c86ba3930a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:11.450405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a42589d-1def-4b0a-afbe-c86ba3930a01 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.451031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a42589d-1def-4b0a-afbe-c86ba3930a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.451682 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a42589d-1def-4b0a-afbe-c86ba3930a01 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:11.453173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823874e7-3234-4e90-b568-f02bfdf7e418 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:11.453810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a42589d-1def-4b0a-afbe-c86ba3930a01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:11.453996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823874e7-3234-4e90-b568-f02bfdf7e418 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.454111 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823874e7-3234-4e90-b568-f02bfdf7e418 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.454426 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823874e7-3234-4e90-b568-f02bfdf7e418 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:11.455986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a42589d-1def-4b0a-afbe-c86ba3930a01 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:11.468478 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823874e7-3234-4e90-b568-f02bfdf7e418 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:11.470412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823874e7-3234-4e90-b568-f02bfdf7e418 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:11.470531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1298/2605] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:11.476830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a42589d-1def-4b0a-afbe-c86ba3930a01 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:11.487447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a42589d-1def-4b0a-afbe-c86ba3930a01 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:11.489048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1308/2606] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:11.499998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a874a8-7e2c-4bde-a779-737a79c0a16a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:11.501058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a874a8-7e2c-4bde-a779-737a79c0a16a admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.501360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a874a8-7e2c-4bde-a779-737a79c0a16a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.501634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a874a8-7e2c-4bde-a779-737a79c0a16a admin admin] PATH_INFO: `/v3/projects/35427f7112df416a9a69a9f9978b5519/users/7a2bffdfc69b48c2845ada640acd218b/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:11.526934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a874a8-7e2c-4bde-a779-737a79c0a16a admin admin] RBAC: Authorizing `identity:create_grant(project_id=35427f7112df416a9a69a9f9978b5519, user_id=7a2bffdfc69b48c2845ada640acd218b, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:11.531772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a874a8-7e2c-4bde-a779-737a79c0a16a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:11.556105 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1299/2607] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:11 2022] PUT /identity/v3/projects/35427f7112df416a9a69a9f9978b5519/users/7a2bffdfc69b48c2845ada640acd218b/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:11.560391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823874e7-3234-4e90-b568-f02bfdf7e418 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:11.576055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823874e7-3234-4e90-b568-f02bfdf7e418 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:11.659549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:11.660484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.660756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.661025 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:11.666107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:11.666107 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:11.685361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1309/2608] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:11.687501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a874a8-7e2c-4bde-a779-737a79c0a16a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:11.698606 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a874a8-7e2c-4bde-a779-737a79c0a16a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:11.707393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:11.707393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.707393 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.708040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] PATH_INFO: `/v3/projects/35427f7112df416a9a69a9f9978b5519/users/7a2bffdfc69b48c2845ada640acd218b/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:11.708040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35427f7112df416a9a69a9f9978b5519, user_id=7a2bffdfc69b48c2845ada640acd218b, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:11.709953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:11.724723 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1300/2609] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:11 2022] PUT /identity/v3/projects/35427f7112df416a9a69a9f9978b5519/users/7a2bffdfc69b48c2845ada640acd218b/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:11.856751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:11.860678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:11.877989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fdd2eb-a901-4b03-97f3-7e8c63997ee6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:11.993732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:11.997896 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:11.998270 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:11.998675 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] PATH_INFO: `/v3/projects/fabbc00fc4634fd9a23d879e64518aed` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:12.001340 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fabbc00fc4634fd9a23d879e64518aed)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:12.004970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:12.047646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:12.090588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fabbc00fc4634fd9a23d879e64518aed', 'request_id': 'req-b72fa86c-0114-4e16-8475-43e3ec75a2e9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.100781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fabbc00fc4634fd9a23d879e64518aed', 'request_id': 'req-b72fa86c-0114-4e16-8475-43e3ec75a2e9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.110144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fabbc00fc4634fd9a23d879e64518aed', 'request_id': 'req-b72fa86c-0114-4e16-8475-43e3ec75a2e9'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.131942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] The token cache is being invalidate because project fabbc00fc4634fd9a23d879e64518aed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:12.132408 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.133498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b72fa86c-0114-4e16-8475-43e3ec75a2e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.136253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1310/2610] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:11 2022] DELETE /identity/v3/projects/fabbc00fc4634fd9a23d879e64518aed => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:12.143267 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:12.146448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:12.167565 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920dd519-4e0b-47ec-834c-64e73b330067 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:12.250073 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:12.251148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:12.251677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:12.251677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] PATH_INFO: `/v3/users/5edff732a0bf4d11be444ae666801854` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:12.253509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5edff732a0bf4d11be444ae666801854)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:12.256004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:12.328269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5edff732a0bf4d11be444ae666801854'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.329408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5edff732a0bf4d11be444ae666801854'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.340127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5edff732a0bf4d11be444ae666801854'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.354444 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5edff732a0bf4d11be444ae666801854'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.372775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5edff732a0bf4d11be444ae666801854'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:12.382408 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1301/2611] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:12 2022] DELETE /identity/v3/users/5edff732a0bf4d11be444ae666801854 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:12.648466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52048fe-7ad6-48ed-a6e4-452cca9d1115 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:12.649722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52048fe-7ad6-48ed-a6e4-452cca9d1115 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:12.650005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52048fe-7ad6-48ed-a6e4-452cca9d1115 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:12.695678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f52048fe-7ad6-48ed-a6e4-452cca9d1115 None None] MFA Rules not processed for user `7a2bffdfc69b48c2845ada640acd218b`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:12.697248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f52048fe-7ad6-48ed-a6e4-452cca9d1115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:12.784008 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:12.795261 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:12.828160 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1311/2612] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:12 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:12.835289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=84,keystone:INSERT=6,keystone:DELETE=10 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:12.845547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52048fe-7ad6-48ed-a6e4-452cca9d1115 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:12.847632 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f52048fe-7ad6-48ed-a6e4-452cca9d1115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:12.941407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736797e3-5afd-40c0-bfbb-a2455907d893 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:12.949566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52048fe-7ad6-48ed-a6e4-452cca9d1115 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:12.950249 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:12.951426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:12.951944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:12.953057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] PATH_INFO: `/v3/projects/4e3663adceb446a4847b125691dbd0a7` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:12.954156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54193389-4296-4fe1-9b2d-a3c5d5787c5b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:12.954841 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54193389-4296-4fe1-9b2d-a3c5d5787c5b service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:12.955153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54193389-4296-4fe1-9b2d-a3c5d5787c5b service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:12.955441 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54193389-4296-4fe1-9b2d-a3c5d5787c5b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:12.955603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e3663adceb446a4847b125691dbd0a7)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:12.957440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-54193389-4296-4fe1-9b2d-a3c5d5787c5b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:12.959314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:13.017574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:13.024064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e3663adceb446a4847b125691dbd0a7', 'request_id': 'req-cd86d49d-4260-47f8-a678-3f829c688a7c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:13.033410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e3663adceb446a4847b125691dbd0a7', 'request_id': 'req-cd86d49d-4260-47f8-a678-3f829c688a7c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:13.038628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54193389-4296-4fe1-9b2d-a3c5d5787c5b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:13.040146 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54193389-4296-4fe1-9b2d-a3c5d5787c5b service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:13.044354 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e3663adceb446a4847b125691dbd0a7', 'request_id': 'req-cd86d49d-4260-47f8-a678-3f829c688a7c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:13.053484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1312/2613] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:12 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:13.060034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] The token cache is being invalidate because project 4e3663adceb446a4847b125691dbd0a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:13.060523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:13.062072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-cd86d49d-4260-47f8-a678-3f829c688a7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:13.065022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1302/2614] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:12 2022] DELETE /identity/v3/projects/4e3663adceb446a4847b125691dbd0a7 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:13.074055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75dbe38-37f1-48e7-a182-106306f648d2 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.076484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75dbe38-37f1-48e7-a182-106306f648d2 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.076796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75dbe38-37f1-48e7-a182-106306f648d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.109883 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f75dbe38-37f1-48e7-a182-106306f648d2 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:13.111195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f75dbe38-37f1-48e7-a182-106306f648d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:13.196505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1313/2615] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:13.209231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cad2b1-ca88-4b19-8983-c24f16d78560 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.209736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cad2b1-ca88-4b19-8983-c24f16d78560 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.210242 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cad2b1-ca88-4b19-8983-c24f16d78560 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.235660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a9cad2b1-ca88-4b19-8983-c24f16d78560 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:13.237217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a9cad2b1-ca88-4b19-8983-c24f16d78560 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:13.245741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1303/2616] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:13.252127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75dbe38-37f1-48e7-a182-106306f648d2 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:13.264706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75dbe38-37f1-48e7-a182-106306f648d2 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:13.275736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:13.276756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.277233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.277636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.283736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:13.284149 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:13.286431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:13.310922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1314/2617] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:13.328651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cad2b1-ca88-4b19-8983-c24f16d78560 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:13.360493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cad2b1-ca88-4b19-8983-c24f16d78560 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:13.383726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239bd958-627b-41f2-9f77-4cc5d0ff7a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:13.384828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239bd958-627b-41f2-9f77-4cc5d0ff7a74 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.385331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239bd958-627b-41f2-9f77-4cc5d0ff7a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.385925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239bd958-627b-41f2-9f77-4cc5d0ff7a74 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.387130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239bd958-627b-41f2-9f77-4cc5d0ff7a74 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:13.392235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239bd958-627b-41f2-9f77-4cc5d0ff7a74 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:13.453505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1304/2618] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:13 2022] POST /identity/v3/projects => generated 401 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:13.459814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:13.480122 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb98ea0-d036-401d-b870-c1d5460fe0c1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:13.551978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb4e9a-1253-4053-bb3f-f8bdd40dbbc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:13.552886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb4e9a-1253-4053-bb3f-f8bdd40dbbc2 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.553297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb4e9a-1253-4053-bb3f-f8bdd40dbbc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.553911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efb4e9a-1253-4053-bb3f-f8bdd40dbbc2 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.555120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efb4e9a-1253-4053-bb3f-f8bdd40dbbc2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:13.557297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efb4e9a-1253-4053-bb3f-f8bdd40dbbc2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:13.594675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239bd958-627b-41f2-9f77-4cc5d0ff7a74 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:13.594957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1315/2619] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:13 2022] POST /identity/v3/users => generated 301 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:13.612421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239bd958-627b-41f2-9f77-4cc5d0ff7a74 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:13.620459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227fac9a-1d48-4129-8d6b-4850aacd6b6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:13.621260 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227fac9a-1d48-4129-8d6b-4850aacd6b6d admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.621532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227fac9a-1d48-4129-8d6b-4850aacd6b6d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.621931 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227fac9a-1d48-4129-8d6b-4850aacd6b6d admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.622981 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227fac9a-1d48-4129-8d6b-4850aacd6b6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:13.625516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227fac9a-1d48-4129-8d6b-4850aacd6b6d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:13.646262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1305/2620] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:13 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:48:13.651737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb4e9a-1253-4053-bb3f-f8bdd40dbbc2 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:13.664903 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efb4e9a-1253-4053-bb3f-f8bdd40dbbc2 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:13.671749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d64638-cde9-4ba9-86ad-2859443831af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:13.672382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d64638-cde9-4ba9-86ad-2859443831af admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.672653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d64638-cde9-4ba9-86ad-2859443831af admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.672927 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d64638-cde9-4ba9-86ad-2859443831af admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.673584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d64638-cde9-4ba9-86ad-2859443831af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:13.675390 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d64638-cde9-4ba9-86ad-2859443831af admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:13.698154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1316/2621] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:13 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:48:13.704026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227fac9a-1d48-4129-8d6b-4850aacd6b6d admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:13.721855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227fac9a-1d48-4129-8d6b-4850aacd6b6d admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:13.729588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861770af-d6c8-4e73-bdb8-a5abbbbda556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:13.730557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861770af-d6c8-4e73-bdb8-a5abbbbda556 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.730879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861770af-d6c8-4e73-bdb8-a5abbbbda556 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.731159 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861770af-d6c8-4e73-bdb8-a5abbbbda556 admin admin] PATH_INFO: `/v3/projects/e384045cd71b4926a8c2f5f21217a485/users/b36178451ac34d1cb1a16f76f286cd5d/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.747560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861770af-d6c8-4e73-bdb8-a5abbbbda556 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e384045cd71b4926a8c2f5f21217a485, user_id=b36178451ac34d1cb1a16f76f286cd5d, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:13.750815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861770af-d6c8-4e73-bdb8-a5abbbbda556 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:13.765899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1306/2622] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:13 2022] PUT /identity/v3/projects/e384045cd71b4926a8c2f5f21217a485/users/b36178451ac34d1cb1a16f76f286cd5d/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:13.777440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b991dd-a5bd-4f6a-9160-0c1e9a79e78b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.777850 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b991dd-a5bd-4f6a-9160-0c1e9a79e78b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.778349 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b991dd-a5bd-4f6a-9160-0c1e9a79e78b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.810796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-37b991dd-a5bd-4f6a-9160-0c1e9a79e78b None None] MFA Rules not processed for user `b36178451ac34d1cb1a16f76f286cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:13.812719 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-37b991dd-a5bd-4f6a-9160-0c1e9a79e78b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:13.869105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:13.896448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1317/2623] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:13 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:13.901878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861770af-d6c8-4e73-bdb8-a5abbbbda556 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:13.957409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:13.981497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861770af-d6c8-4e73-bdb8-a5abbbbda556 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:13.987005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7d4bb4-1d3b-4555-a04f-5d5b3281ee7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:13.987610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7d4bb4-1d3b-4555-a04f-5d5b3281ee7c admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:13.987892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7d4bb4-1d3b-4555-a04f-5d5b3281ee7c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:13.988160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7d4bb4-1d3b-4555-a04f-5d5b3281ee7c admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:13.988797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7d4bb4-1d3b-4555-a04f-5d5b3281ee7c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:13.990695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7d4bb4-1d3b-4555-a04f-5d5b3281ee7c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.020142 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1307/2624] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:13 2022] POST /identity/v3/projects => generated 401 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:14.024320 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b991dd-a5bd-4f6a-9160-0c1e9a79e78b None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:14.038004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b991dd-a5bd-4f6a-9160-0c1e9a79e78b None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:14.125120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a391fbd-ca53-4300-8d10-80f0146a830b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:14.125834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a391fbd-ca53-4300-8d10-80f0146a830b admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.126238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a391fbd-ca53-4300-8d10-80f0146a830b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.126545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a391fbd-ca53-4300-8d10-80f0146a830b admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.127237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a391fbd-ca53-4300-8d10-80f0146a830b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:14.129164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a391fbd-ca53-4300-8d10-80f0146a830b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.153227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1318/2625] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:14 2022] POST /identity/v3/users => generated 307 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:14.158631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7d4bb4-1d3b-4555-a04f-5d5b3281ee7c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:14.169907 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7d4bb4-1d3b-4555-a04f-5d5b3281ee7c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:14.176771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb1c2dd-851d-433e-9b0e-d6401d07784f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:14.177862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1c2dd-851d-433e-9b0e-d6401d07784f admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.178367 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1c2dd-851d-433e-9b0e-d6401d07784f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.178919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1c2dd-851d-433e-9b0e-d6401d07784f admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.180130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb1c2dd-851d-433e-9b0e-d6401d07784f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:14.183541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb1c2dd-851d-433e-9b0e-d6401d07784f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.199349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1308/2626] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:48:14.204356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a391fbd-ca53-4300-8d10-80f0146a830b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:14.215601 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a391fbd-ca53-4300-8d10-80f0146a830b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:14.221521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37732e67-0bb5-42bf-835c-b43150163c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:14.222355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37732e67-0bb5-42bf-835c-b43150163c14 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.222638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37732e67-0bb5-42bf-835c-b43150163c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.222955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37732e67-0bb5-42bf-835c-b43150163c14 admin admin] PATH_INFO: `/v3/projects/999387dbb9ad4bd2925b8230ccfa7009/users/36d2022c22e14b808b8d723010dcf87a/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.247541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37732e67-0bb5-42bf-835c-b43150163c14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=999387dbb9ad4bd2925b8230ccfa7009, user_id=36d2022c22e14b808b8d723010dcf87a, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:14.250267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37732e67-0bb5-42bf-835c-b43150163c14 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.266892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1319/2627] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:14 2022] PUT /identity/v3/projects/999387dbb9ad4bd2925b8230ccfa7009/users/36d2022c22e14b808b8d723010dcf87a/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:14.271843 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb1c2dd-851d-433e-9b0e-d6401d07784f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:14.285653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb1c2dd-851d-433e-9b0e-d6401d07784f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:14.352783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78bc9de-9062-431f-b4ef-1c7eb3a73866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:14.353594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78bc9de-9062-431f-b4ef-1c7eb3a73866 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.353740 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78bc9de-9062-431f-b4ef-1c7eb3a73866 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.353899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78bc9de-9062-431f-b4ef-1c7eb3a73866 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.354576 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78bc9de-9062-431f-b4ef-1c7eb3a73866 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:14.356382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78bc9de-9062-431f-b4ef-1c7eb3a73866 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.367448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1309/2628] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:14.372040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37732e67-0bb5-42bf-835c-b43150163c14 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:14.384542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37732e67-0bb5-42bf-835c-b43150163c14 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:14.389617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:14.390812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.391090 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.391360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] PATH_INFO: `/v3/projects/999387dbb9ad4bd2925b8230ccfa7009/users/36d2022c22e14b808b8d723010dcf87a/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.393321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] RBAC: Authorizing `identity:create_grant(project_id=999387dbb9ad4bd2925b8230ccfa7009, user_id=36d2022c22e14b808b8d723010dcf87a, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:14.396224 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.411350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1320/2629] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:14 2022] PUT /identity/v3/projects/999387dbb9ad4bd2925b8230ccfa7009/users/36d2022c22e14b808b8d723010dcf87a/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:14.485390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b198723-d748-463d-90f6-07078bdea4d5 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.485691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b198723-d748-463d-90f6-07078bdea4d5 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.485972 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b198723-d748-463d-90f6-07078bdea4d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.526697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7b198723-d748-463d-90f6-07078bdea4d5 None None] MFA Rules not processed for user `36d2022c22e14b808b8d723010dcf87a`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:14.528201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7b198723-d748-463d-90f6-07078bdea4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:14.562307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:14.564565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:14.635030 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac53fcec-86e5-486e-a45f-e2a9c8c68881 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:14.636891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1310/2630] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:14 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:14.638894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:14.639968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.640807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.641151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.643591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:14.653214 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b198723-d748-463d-90f6-07078bdea4d5 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:14.658302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7b198723-d748-463d-90f6-07078bdea4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:14.674092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b198723-d748-463d-90f6-07078bdea4d5 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:14.678945 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c915246b-5318-4c3f-b290-33d89d5242b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:14.679605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915246b-5318-4c3f-b290-33d89d5242b6 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.679882 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915246b-5318-4c3f-b290-33d89d5242b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.680154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915246b-5318-4c3f-b290-33d89d5242b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.694647 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c915246b-5318-4c3f-b290-33d89d5242b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:14.697510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c915246b-5318-4c3f-b290-33d89d5242b6 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.708947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1311/2631] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:14 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:14.714153 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:14.743495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:14.746360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.760394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1321/2632] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:14 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:14.846048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c915246b-5318-4c3f-b290-33d89d5242b6 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:14.876023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c915246b-5318-4c3f-b290-33d89d5242b6 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:14.880675 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e0b3f0-e540-4ff6-86a6-b2573cf60c42 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:14.881322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e0b3f0-e540-4ff6-86a6-b2573cf60c42 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:14.881600 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e0b3f0-e540-4ff6-86a6-b2573cf60c42 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:14.881950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e0b3f0-e540-4ff6-86a6-b2573cf60c42 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:14.908641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:14.959246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e0b3f0-e540-4ff6-86a6-b2573cf60c42 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:14.961495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e0b3f0-e540-4ff6-86a6-b2573cf60c42 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:14.974423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1312/2633] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:14 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:19.438550 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:19.518430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dae4d6-80bb-4808-9a99-38a8c08e6d05 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:19.523669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:19.524449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:19.524732 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:19.525007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] PATH_INFO: `/v3/users/b36178451ac34d1cb1a16f76f286cd5d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:19.526221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] RBAC: Authorizing `identity:delete_user(user_id=b36178451ac34d1cb1a16f76f286cd5d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:19.528273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:19.583374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b36178451ac34d1cb1a16f76f286cd5d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:19.597592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b36178451ac34d1cb1a16f76f286cd5d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:19.598808 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b36178451ac34d1cb1a16f76f286cd5d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:19.599802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b36178451ac34d1cb1a16f76f286cd5d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:19.612258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b36178451ac34d1cb1a16f76f286cd5d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:19.645510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1322/2634] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:19 2022] DELETE /identity/v3/users/b36178451ac34d1cb1a16f76f286cd5d => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:19.652850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e47b297-d616-497a-a924-5db3e1c621a4 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:19.653415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e47b297-d616-497a-a924-5db3e1c621a4 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:19.653415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e47b297-d616-497a-a924-5db3e1c621a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:19.676500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9e47b297-d616-497a-a924-5db3e1c621a4 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:19.677816 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9e47b297-d616-497a-a924-5db3e1c621a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:19.765923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1313/2635] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:19.782238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:19.783812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:19.868923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459b8ea0-ae07-4d89-9c44-8641563606db admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:19.872670 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b24c2e-d3e3-4c0f-a1b8-12bbe6c0ff43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:19.873288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b24c2e-d3e3-4c0f-a1b8-12bbe6c0ff43 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:19.873570 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b24c2e-d3e3-4c0f-a1b8-12bbe6c0ff43 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:19.873872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b24c2e-d3e3-4c0f-a1b8-12bbe6c0ff43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:19.886083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b24c2e-d3e3-4c0f-a1b8-12bbe6c0ff43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:19.888920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b24c2e-d3e3-4c0f-a1b8-12bbe6c0ff43 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:19.907559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1323/2636] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:20.282689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e47b297-d616-497a-a924-5db3e1c621a4 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:20.284454 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9e47b297-d616-497a-a924-5db3e1c621a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:20.312893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e47b297-d616-497a-a924-5db3e1c621a4 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:20.322345 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:20.323307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:20.323482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:20.323789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] PATH_INFO: `/v3/projects/e384045cd71b4926a8c2f5f21217a485` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:20.326454 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e384045cd71b4926a8c2f5f21217a485)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:20.328930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:20.403712 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e384045cd71b4926a8c2f5f21217a485', 'request_id': 'req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.413850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e384045cd71b4926a8c2f5f21217a485', 'request_id': 'req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.422646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e384045cd71b4926a8c2f5f21217a485', 'request_id': 'req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.438305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] The token cache is being invalidate because project e384045cd71b4926a8c2f5f21217a485 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:20.438840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.439926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.442877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1314/2637] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:20 2022] DELETE /identity/v3/projects/e384045cd71b4926a8c2f5f21217a485 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:20.449613 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b24c2e-d3e3-4c0f-a1b8-12bbe6c0ff43 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:20.458110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:20.463299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b24c2e-d3e3-4c0f-a1b8-12bbe6c0ff43 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:20.541423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:20.542442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:20.542765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:20.543055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] PATH_INFO: `/v3/users/36d2022c22e14b808b8d723010dcf87a` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:20.544248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=36d2022c22e14b808b8d723010dcf87a)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:20.546241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:20.614524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36d2022c22e14b808b8d723010dcf87a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.622155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36d2022c22e14b808b8d723010dcf87a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.622931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36d2022c22e14b808b8d723010dcf87a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.623597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36d2022c22e14b808b8d723010dcf87a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.635338 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-72323989-597b-4ede-b037-b138a4c8cf9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36d2022c22e14b808b8d723010dcf87a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:20.657280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1324/2638] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:20 2022] DELETE /identity/v3/users/36d2022c22e14b808b8d723010dcf87a => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:21.033441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:21.035244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:21.052126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d510afa-b0d9-4e17-b3ce-3be2951c1e85 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:21.129241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:21.151449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:21.152176 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.152502 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.152823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] PATH_INFO: `/v3/projects/999387dbb9ad4bd2925b8230ccfa7009` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.154066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=999387dbb9ad4bd2925b8230ccfa7009)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:21.156767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:21.230163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '999387dbb9ad4bd2925b8230ccfa7009', 'request_id': 'req-5fbb1582-c941-43bc-bb14-917d8d3dca80'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:21.242383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '999387dbb9ad4bd2925b8230ccfa7009', 'request_id': 'req-5fbb1582-c941-43bc-bb14-917d8d3dca80'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:21.252086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '999387dbb9ad4bd2925b8230ccfa7009', 'request_id': 'req-5fbb1582-c941-43bc-bb14-917d8d3dca80'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:21.264437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] The token cache is being invalidate because project 999387dbb9ad4bd2925b8230ccfa7009 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:21.265277 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:21.266026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5fbb1582-c941-43bc-bb14-917d8d3dca80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:21.267672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1315/2639] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:21 2022] DELETE /identity/v3/projects/999387dbb9ad4bd2925b8230ccfa7009 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:21.276102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83392a4f-ef0d-4649-bb5d-5113d0568b32 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.276598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83392a4f-ef0d-4649-bb5d-5113d0568b32 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.277080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83392a4f-ef0d-4649-bb5d-5113d0568b32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.299034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-83392a4f-ef0d-4649-bb5d-5113d0568b32 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:21.300308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-83392a4f-ef0d-4649-bb5d-5113d0568b32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:21.394806 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1325/2640] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:21.403185 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a05203-8c61-4711-873b-9dd1beb23349 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.403693 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a05203-8c61-4711-873b-9dd1beb23349 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.404173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a05203-8c61-4711-873b-9dd1beb23349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.430478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-25a05203-8c61-4711-873b-9dd1beb23349 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:21.431759 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-25a05203-8c61-4711-873b-9dd1beb23349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:21.444040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1316/2641] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:21.449316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83392a4f-ef0d-4649-bb5d-5113d0568b32 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:21.462260 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83392a4f-ef0d-4649-bb5d-5113d0568b32 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:21.467396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:21.468273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.468556 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.468865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.469520 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:21.469823 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:21.471470 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:21.487758 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1326/2642] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:21.492717 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a05203-8c61-4711-873b-9dd1beb23349 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:21.503111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a05203-8c61-4711-873b-9dd1beb23349 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:21.509406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459e0b31-74e8-4360-9604-267787255daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:21.510213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459e0b31-74e8-4360-9604-267787255daf admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.510621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459e0b31-74e8-4360-9604-267787255daf admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.511023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459e0b31-74e8-4360-9604-267787255daf admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.511923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459e0b31-74e8-4360-9604-267787255daf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:21.514404 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459e0b31-74e8-4360-9604-267787255daf admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:21.557177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1317/2643] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:21 2022] POST /identity/v3/projects => generated 389 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:21.563695 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:21.575063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128bf5d7-19e1-42f2-858f-fe2139ce2db3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:21.584027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:21.665278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0500c128-0099-4ddd-a9c6-2fa759e64fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:21.665278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500c128-0099-4ddd-a9c6-2fa759e64fe6 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.665725 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500c128-0099-4ddd-a9c6-2fa759e64fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.665997 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500c128-0099-4ddd-a9c6-2fa759e64fe6 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.666662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0500c128-0099-4ddd-a9c6-2fa759e64fe6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:21.668699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0500c128-0099-4ddd-a9c6-2fa759e64fe6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:21.703534 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1327/2644] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:21 2022] POST /identity/v3/users => generated 295 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:21.709135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459e0b31-74e8-4360-9604-267787255daf admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:21.726928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459e0b31-74e8-4360-9604-267787255daf admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:21.735440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df63dc59-0141-487e-bd46-8caa2c1cae01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:21.736927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df63dc59-0141-487e-bd46-8caa2c1cae01 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.737394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df63dc59-0141-487e-bd46-8caa2c1cae01 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.737916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df63dc59-0141-487e-bd46-8caa2c1cae01 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.739141 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df63dc59-0141-487e-bd46-8caa2c1cae01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:21.741750 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df63dc59-0141-487e-bd46-8caa2c1cae01 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:21.760810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1318/2645] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:21 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:21.766342 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0500c128-0099-4ddd-a9c6-2fa759e64fe6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:21.779595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0500c128-0099-4ddd-a9c6-2fa759e64fe6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:21.784953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb109b8-edf4-4b30-9c4f-ae634e05086f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:21.785884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb109b8-edf4-4b30-9c4f-ae634e05086f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.786244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb109b8-edf4-4b30-9c4f-ae634e05086f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.786628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb109b8-edf4-4b30-9c4f-ae634e05086f admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.787608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb109b8-edf4-4b30-9c4f-ae634e05086f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:21.789325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb109b8-edf4-4b30-9c4f-ae634e05086f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:21.803778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1328/2646] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:48:21.808953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df63dc59-0141-487e-bd46-8caa2c1cae01 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:21.833465 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df63dc59-0141-487e-bd46-8caa2c1cae01 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:21.842072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:21.844037 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.844436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.844830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] PATH_INFO: `/v3/projects/0084e9c31be24b248672a312e039b68c/users/b191436341a9487d9d9b80f04835522a/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.870518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0084e9c31be24b248672a312e039b68c, user_id=b191436341a9487d9d9b80f04835522a, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:21.874091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:21.891747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1319/2647] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:21 2022] PUT /identity/v3/projects/0084e9c31be24b248672a312e039b68c/users/b191436341a9487d9d9b80f04835522a/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:21.912381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1421406f-d8e7-4883-b2eb-c09533287bb4 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:21.913034 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1421406f-d8e7-4883-b2eb-c09533287bb4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:21.913382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1421406f-d8e7-4883-b2eb-c09533287bb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:21.993132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1421406f-d8e7-4883-b2eb-c09533287bb4 None None] MFA Rules not processed for user `b191436341a9487d9d9b80f04835522a`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:21.994577 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1421406f-d8e7-4883-b2eb-c09533287bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:22.103974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1329/2648] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:21 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:22.122433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:22.126216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:22.209610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c7912b-9703-46fb-b7fd-284240cb1093 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:22.213617 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09857afd-94ad-4efa-8fa9-e2c8430a4def service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:22.214232 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09857afd-94ad-4efa-8fa9-e2c8430a4def service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:22.214505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09857afd-94ad-4efa-8fa9-e2c8430a4def service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:22.214799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09857afd-94ad-4efa-8fa9-e2c8430a4def service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:22.226615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09857afd-94ad-4efa-8fa9-e2c8430a4def service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:22.229346 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09857afd-94ad-4efa-8fa9-e2c8430a4def service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:22.241160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1320/2649] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:22 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:27.922962 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1421406f-d8e7-4883-b2eb-c09533287bb4 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:27.924576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1421406f-d8e7-4883-b2eb-c09533287bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:28.001271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1421406f-d8e7-4883-b2eb-c09533287bb4 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:28.006766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:28.007529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:28.007821 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:28.008116 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] PATH_INFO: `/v3/users/aea0676377ae4d95991a7cf766bd6cce` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:28.009678 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aea0676377ae4d95991a7cf766bd6cce)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:28.011656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:28.041378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:28.098942 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aea0676377ae4d95991a7cf766bd6cce'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:28.106460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aea0676377ae4d95991a7cf766bd6cce'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:28.107240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aea0676377ae4d95991a7cf766bd6cce'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:28.107924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aea0676377ae4d95991a7cf766bd6cce'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:28.116744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aea0676377ae4d95991a7cf766bd6cce'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:28.153531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1330/2650] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:27 2022] DELETE /identity/v3/users/aea0676377ae4d95991a7cf766bd6cce => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:28.158146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be364b71-89b1-4919-aa9b-844d6f16fa65 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:28.158679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be364b71-89b1-4919-aa9b-844d6f16fa65 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:28.158679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be364b71-89b1-4919-aa9b-844d6f16fa65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:28.188039 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-be364b71-89b1-4919-aa9b-844d6f16fa65 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:28.189496 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-be364b71-89b1-4919-aa9b-844d6f16fa65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:28.200492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:28.301131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1321/2651] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:28.316522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:28.318063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:28.411244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abc230e-da9f-43bd-a97b-285e96416348 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:28.418236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3045bb-3cde-4d6a-9dfb-c40bbe3e466f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:28.424502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3045bb-3cde-4d6a-9dfb-c40bbe3e466f service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:28.424502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3045bb-3cde-4d6a-9dfb-c40bbe3e466f service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:28.424502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3045bb-3cde-4d6a-9dfb-c40bbe3e466f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:28.433264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3045bb-3cde-4d6a-9dfb-c40bbe3e466f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:28.435248 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3045bb-3cde-4d6a-9dfb-c40bbe3e466f service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:28.453312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1331/2652] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:28.997357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be364b71-89b1-4919-aa9b-844d6f16fa65 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:28.999014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-be364b71-89b1-4919-aa9b-844d6f16fa65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:29.024801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be364b71-89b1-4919-aa9b-844d6f16fa65 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:29.028769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:29.029459 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.030002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.030298 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] PATH_INFO: `/v3/projects/4462ca02650f47d4bcd2a74789260a61` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.031883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4462ca02650f47d4bcd2a74789260a61)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:29.033910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:29.118558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4462ca02650f47d4bcd2a74789260a61', 'request_id': 'req-59bfa1bc-7189-4dcf-be57-401e242575e7'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:29.122473 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4462ca02650f47d4bcd2a74789260a61', 'request_id': 'req-59bfa1bc-7189-4dcf-be57-401e242575e7'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:29.135018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4462ca02650f47d4bcd2a74789260a61', 'request_id': 'req-59bfa1bc-7189-4dcf-be57-401e242575e7'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:29.154471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] The token cache is being invalidate because project 4462ca02650f47d4bcd2a74789260a61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:29.155238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:29.156826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59bfa1bc-7189-4dcf-be57-401e242575e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:29.158651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1322/2653] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:28 2022] DELETE /identity/v3/projects/4462ca02650f47d4bcd2a74789260a61 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:29.163955 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ad79f5-4ff0-426b-87f7-777541f2316f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.164965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ad79f5-4ff0-426b-87f7-777541f2316f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.165674 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ad79f5-4ff0-426b-87f7-777541f2316f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.204891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e5ad79f5-4ff0-426b-87f7-777541f2316f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:29.206421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e5ad79f5-4ff0-426b-87f7-777541f2316f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:29.306073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1332/2654] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:29.315573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5941a43a-b4b9-45d5-953b-6401049243bc None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.315884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5941a43a-b4b9-45d5-953b-6401049243bc None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.316424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5941a43a-b4b9-45d5-953b-6401049243bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.345806 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5941a43a-b4b9-45d5-953b-6401049243bc None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:29.347201 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5941a43a-b4b9-45d5-953b-6401049243bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:29.356178 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1323/2655] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:29.363310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ad79f5-4ff0-426b-87f7-777541f2316f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:29.375280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ad79f5-4ff0-426b-87f7-777541f2316f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:29.380602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:29.381429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.381741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.382086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.382867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:29.383180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:29.385194 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:29.406199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1333/2656] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:29.408862 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5941a43a-b4b9-45d5-953b-6401049243bc None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:29.418982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5941a43a-b4b9-45d5-953b-6401049243bc None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:29.426129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc006e6-3245-4301-8cd1-e2a63ef71c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:29.427106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc006e6-3245-4301-8cd1-e2a63ef71c4f admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.427585 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc006e6-3245-4301-8cd1-e2a63ef71c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.428023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc006e6-3245-4301-8cd1-e2a63ef71c4f admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.428694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc006e6-3245-4301-8cd1-e2a63ef71c4f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:29.431548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc006e6-3245-4301-8cd1-e2a63ef71c4f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:29.471326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1324/2657] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:29 2022] POST /identity/v3/projects => generated 383 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:29.473561 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:29.485231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ead260-1e7a-4558-b488-5ba5b704b1da admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:29.533393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:29.577938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc5991-732d-455d-935f-5fc0716ac697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:29.578761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc5991-732d-455d-935f-5fc0716ac697 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.579024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc5991-732d-455d-935f-5fc0716ac697 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.579303 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc5991-732d-455d-935f-5fc0716ac697 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.579970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc5991-732d-455d-935f-5fc0716ac697 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:29.582035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc5991-732d-455d-935f-5fc0716ac697 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:29.608614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1334/2658] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:29 2022] POST /identity/v3/users => generated 292 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:29.613203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc006e6-3245-4301-8cd1-e2a63ef71c4f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:29.625546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc006e6-3245-4301-8cd1-e2a63ef71c4f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:29.630832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899f5cbb-48af-40f8-a22e-071f3ace9456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:29.631460 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f5cbb-48af-40f8-a22e-071f3ace9456 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.631741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f5cbb-48af-40f8-a22e-071f3ace9456 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.632024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f5cbb-48af-40f8-a22e-071f3ace9456 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.632730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899f5cbb-48af-40f8-a22e-071f3ace9456 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:29.634529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899f5cbb-48af-40f8-a22e-071f3ace9456 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:29.650268 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1325/2659] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:29.651956 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc5991-732d-455d-935f-5fc0716ac697 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:29.665952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc5991-732d-455d-935f-5fc0716ac697 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:29.671200 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a1c6ff-0d64-44e4-97bc-d9f70005e982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:29.672269 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a1c6ff-0d64-44e4-97bc-d9f70005e982 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.672564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a1c6ff-0d64-44e4-97bc-d9f70005e982 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.672857 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a1c6ff-0d64-44e4-97bc-d9f70005e982 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.673574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a1c6ff-0d64-44e4-97bc-d9f70005e982 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:29.675504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a1c6ff-0d64-44e4-97bc-d9f70005e982 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:29.688568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1335/2660] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:29 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:29.692909 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899f5cbb-48af-40f8-a22e-071f3ace9456 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:29.707288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899f5cbb-48af-40f8-a22e-071f3ace9456 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:29.713292 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868cec02-e04a-4891-b7a8-776f086b710c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:29.713974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868cec02-e04a-4891-b7a8-776f086b710c admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.714253 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868cec02-e04a-4891-b7a8-776f086b710c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.714946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868cec02-e04a-4891-b7a8-776f086b710c admin admin] PATH_INFO: `/v3/projects/f37dab88a82543b78761778a8e6e6834/users/7fcc46dae42f46bdbbfd076a6cd23434/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.732486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868cec02-e04a-4891-b7a8-776f086b710c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f37dab88a82543b78761778a8e6e6834, user_id=7fcc46dae42f46bdbbfd076a6cd23434, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:29.735831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868cec02-e04a-4891-b7a8-776f086b710c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:29.754850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1326/2661] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:29 2022] PUT /identity/v3/projects/f37dab88a82543b78761778a8e6e6834/users/7fcc46dae42f46bdbbfd076a6cd23434/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:29.768661 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab01dc-2ee0-4646-af97-b736a70ecb23 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:29.769142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab01dc-2ee0-4646-af97-b736a70ecb23 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:29.769499 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab01dc-2ee0-4646-af97-b736a70ecb23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:29.802815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5dab01dc-2ee0-4646-af97-b736a70ecb23 None None] MFA Rules not processed for user `7fcc46dae42f46bdbbfd076a6cd23434`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:29.804144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5dab01dc-2ee0-4646-af97-b736a70ecb23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:29.891356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1336/2662] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:29.899421 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868cec02-e04a-4891-b7a8-776f086b710c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:29.959004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:30.000500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868cec02-e04a-4891-b7a8-776f086b710c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.006885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc96cbe6-1810-4d27-bbae-78ec908b7208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.007566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc96cbe6-1810-4d27-bbae-78ec908b7208 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.007819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc96cbe6-1810-4d27-bbae-78ec908b7208 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.008128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc96cbe6-1810-4d27-bbae-78ec908b7208 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.009111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc96cbe6-1810-4d27-bbae-78ec908b7208 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.011274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc96cbe6-1810-4d27-bbae-78ec908b7208 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.050854 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1327/2663] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:29 2022] POST /identity/v3/projects => generated 385 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:30.055180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dab01dc-2ee0-4646-af97-b736a70ecb23 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.067452 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dab01dc-2ee0-4646-af97-b736a70ecb23 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.144923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edba45-c941-4d7b-9662-9e8b1026053a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.145568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edba45-c941-4d7b-9662-9e8b1026053a admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.145854 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edba45-c941-4d7b-9662-9e8b1026053a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.146131 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23edba45-c941-4d7b-9662-9e8b1026053a admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.147049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23edba45-c941-4d7b-9662-9e8b1026053a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.149372 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23edba45-c941-4d7b-9662-9e8b1026053a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.173019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1337/2664] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:30 2022] POST /identity/v3/users => generated 293 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:30.179029 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc96cbe6-1810-4d27-bbae-78ec908b7208 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.191144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc96cbe6-1810-4d27-bbae-78ec908b7208 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.196206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11d7a5-0dd3-4ed7-a209-563688baeb93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.196828 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11d7a5-0dd3-4ed7-a209-563688baeb93 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.197434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11d7a5-0dd3-4ed7-a209-563688baeb93 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.197434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff11d7a5-0dd3-4ed7-a209-563688baeb93 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.198018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff11d7a5-0dd3-4ed7-a209-563688baeb93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.199656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff11d7a5-0dd3-4ed7-a209-563688baeb93 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.213436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1328/2665] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:30.217714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edba45-c941-4d7b-9662-9e8b1026053a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.230756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23edba45-c941-4d7b-9662-9e8b1026053a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.236007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a606d8a-dbad-458b-bea2-49fc825779ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.236717 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a606d8a-dbad-458b-bea2-49fc825779ff admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.236996 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a606d8a-dbad-458b-bea2-49fc825779ff admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.237282 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a606d8a-dbad-458b-bea2-49fc825779ff admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.237936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a606d8a-dbad-458b-bea2-49fc825779ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.239798 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a606d8a-dbad-458b-bea2-49fc825779ff admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.259316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1338/2666] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:30.263944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11d7a5-0dd3-4ed7-a209-563688baeb93 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.281092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff11d7a5-0dd3-4ed7-a209-563688baeb93 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.287629 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be0546a-6f94-425e-b919-d854929f35d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.288633 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be0546a-6f94-425e-b919-d854929f35d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.289016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be0546a-6f94-425e-b919-d854929f35d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.289423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be0546a-6f94-425e-b919-d854929f35d4 admin admin] PATH_INFO: `/v3/projects/baeec031611e45febe34e52feb2fa49f/users/0aebb188b0f24c6293c3b2dea8ec8382/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.310331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be0546a-6f94-425e-b919-d854929f35d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=baeec031611e45febe34e52feb2fa49f, user_id=0aebb188b0f24c6293c3b2dea8ec8382, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.313915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be0546a-6f94-425e-b919-d854929f35d4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.330317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1329/2667] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:30 2022] PUT /identity/v3/projects/baeec031611e45febe34e52feb2fa49f/users/0aebb188b0f24c6293c3b2dea8ec8382/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:30.342407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c82803-5401-49db-84cc-ed50c2a07bf4 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.342756 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c82803-5401-49db-84cc-ed50c2a07bf4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.343401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c82803-5401-49db-84cc-ed50c2a07bf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.379132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-d1c82803-5401-49db-84cc-ed50c2a07bf4 None None] MFA Rules not processed for user `0aebb188b0f24c6293c3b2dea8ec8382`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:30.380429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d1c82803-5401-49db-84cc-ed50c2a07bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:30.384792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:30.470785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1339/2668] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:30.475561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be0546a-6f94-425e-b919-d854929f35d4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.549307 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be0546a-6f94-425e-b919-d854929f35d4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.554320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b076204-34d0-48b8-98e4-a5bba36393aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.554965 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b076204-34d0-48b8-98e4-a5bba36393aa admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.555246 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b076204-34d0-48b8-98e4-a5bba36393aa admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.555515 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b076204-34d0-48b8-98e4-a5bba36393aa admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.556164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b076204-34d0-48b8-98e4-a5bba36393aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.558115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b076204-34d0-48b8-98e4-a5bba36393aa admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.596313 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1330/2669] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:30 2022] POST /identity/v3/projects => generated 385 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:30.601884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c82803-5401-49db-84cc-ed50c2a07bf4 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.614902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c82803-5401-49db-84cc-ed50c2a07bf4 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.690852 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492f3f54-cded-425e-82ca-13275bbf0ef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.691493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492f3f54-cded-425e-82ca-13275bbf0ef6 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.691773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492f3f54-cded-425e-82ca-13275bbf0ef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.692049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492f3f54-cded-425e-82ca-13275bbf0ef6 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.692706 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492f3f54-cded-425e-82ca-13275bbf0ef6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.694662 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492f3f54-cded-425e-82ca-13275bbf0ef6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.717640 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1340/2670] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:30 2022] POST /identity/v3/users => generated 299 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:30.722000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b076204-34d0-48b8-98e4-a5bba36393aa admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.734094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b076204-34d0-48b8-98e4-a5bba36393aa admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.739591 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd0de78-5f30-4664-994d-b5bfd2949b9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.740314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0de78-5f30-4664-994d-b5bfd2949b9a admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.740596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0de78-5f30-4664-994d-b5bfd2949b9a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.740865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0de78-5f30-4664-994d-b5bfd2949b9a admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.741517 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd0de78-5f30-4664-994d-b5bfd2949b9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.743184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd0de78-5f30-4664-994d-b5bfd2949b9a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.755164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1331/2671] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:30.759535 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492f3f54-cded-425e-82ca-13275bbf0ef6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.770304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492f3f54-cded-425e-82ca-13275bbf0ef6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.775938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d32882-f4ec-4228-a838-8ed236526ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.776680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d32882-f4ec-4228-a838-8ed236526ead admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.777016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d32882-f4ec-4228-a838-8ed236526ead admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.777345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d32882-f4ec-4228-a838-8ed236526ead admin admin] PATH_INFO: `/v3/projects/b13e875bc4684a0ba7b5466dfc88deb5/users/f4b42133fdf74666ac54d8a4e2a8cd90/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.802217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d32882-f4ec-4228-a838-8ed236526ead admin admin] RBAC: Authorizing `identity:create_grant(project_id=b13e875bc4684a0ba7b5466dfc88deb5, user_id=f4b42133fdf74666ac54d8a4e2a8cd90, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.804922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d32882-f4ec-4228-a838-8ed236526ead admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.836930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1341/2672] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:30 2022] PUT /identity/v3/projects/b13e875bc4684a0ba7b5466dfc88deb5/users/f4b42133fdf74666ac54d8a4e2a8cd90/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:30.841587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd0de78-5f30-4664-994d-b5bfd2949b9a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.854876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd0de78-5f30-4664-994d-b5bfd2949b9a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.921902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d31080-c8ca-4397-957d-979f684ecd59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.922518 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d31080-c8ca-4397-957d-979f684ecd59 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.922834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d31080-c8ca-4397-957d-979f684ecd59 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.923115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d31080-c8ca-4397-957d-979f684ecd59 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.923910 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d31080-c8ca-4397-957d-979f684ecd59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.925578 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d31080-c8ca-4397-957d-979f684ecd59 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.927258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:30.941198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1332/2673] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:30.944436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d32882-f4ec-4228-a838-8ed236526ead admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:30.956083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d32882-f4ec-4228-a838-8ed236526ead admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:30.963667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:30.964343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:30.964620 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:30.964891 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] PATH_INFO: `/v3/projects/b13e875bc4684a0ba7b5466dfc88deb5/users/f4b42133fdf74666ac54d8a4e2a8cd90/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:30.967533 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b13e875bc4684a0ba7b5466dfc88deb5, user_id=f4b42133fdf74666ac54d8a4e2a8cd90, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:30.970547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:30.982343 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1342/2674] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:30 2022] PUT /identity/v3/projects/b13e875bc4684a0ba7b5466dfc88deb5/users/f4b42133fdf74666ac54d8a4e2a8cd90/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:31.002140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5169b9-6430-40cd-a9ed-1d80be450196 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:31.003614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5169b9-6430-40cd-a9ed-1d80be450196 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:31.004061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5169b9-6430-40cd-a9ed-1d80be450196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:31.046455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2c5169b9-6430-40cd-a9ed-1d80be450196 None None] MFA Rules not processed for user `f4b42133fdf74666ac54d8a4e2a8cd90`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:31.048387 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2c5169b9-6430-40cd-a9ed-1d80be450196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:31.270394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1333/2675] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:30 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:31.287257 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:31.289014 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:31.355391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422ca9b6-7999-47a6-a2d2-7e6f54b3bc20 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:31.360865 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e357ac83-4af0-4780-a596-d342485d11bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:31.361490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e357ac83-4af0-4780-a596-d342485d11bc service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:31.361769 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e357ac83-4af0-4780-a596-d342485d11bc service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:31.362041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e357ac83-4af0-4780-a596-d342485d11bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:31.373159 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:31.429484 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e357ac83-4af0-4780-a596-d342485d11bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:31.430984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e357ac83-4af0-4780-a596-d342485d11bc service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:31.441493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1343/2676] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:33.106048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5169b9-6430-40cd-a9ed-1d80be450196 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:33.108281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2c5169b9-6430-40cd-a9ed-1d80be450196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:33.128447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5169b9-6430-40cd-a9ed-1d80be450196 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:33.134797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:33.135495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:33.135842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:33.136121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:33.224494 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:33.227071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:33.240263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1334/2677] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:36.060455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e357ac83-4af0-4780-a596-d342485d11bc service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:36.073141 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e357ac83-4af0-4780-a596-d342485d11bc service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:36.080076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:36.081100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:36.081588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:36.082088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:36.100495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:36.102118 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:36.116970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1344/2678] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:36 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:38.019846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:38.023035 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:38.118480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f81d82a-a314-4a8a-afad-3aef6ca22913 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:38.122559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:38.123813 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:38.124091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:38.124586 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:38.127160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:38.170857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:38.220772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:38.222349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:38.236671 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1335/2679] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:38.570607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:38.572255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:38.597406 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f8192-1610-4c42-9fcc-e6c43c27a973 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:38.602524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:38.603228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:38.603505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:38.604059 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] PATH_INFO: `/v3/users/662c210639dd4065ae46b18fd2737e53` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:38.604935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=662c210639dd4065ae46b18fd2737e53)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:38.607002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:38.675218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '662c210639dd4065ae46b18fd2737e53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:38.685325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '662c210639dd4065ae46b18fd2737e53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:38.686948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '662c210639dd4065ae46b18fd2737e53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:38.687843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '662c210639dd4065ae46b18fd2737e53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:38.702802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '662c210639dd4065ae46b18fd2737e53'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:38.738394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1345/2680] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:38 2022] DELETE /identity/v3/users/662c210639dd4065ae46b18fd2737e53 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:38.991498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:39.005013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519b20a8-dade-453b-a838-fda8d40f0353 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:39.096855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:39.098374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:39.098840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:39.099170 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] PATH_INFO: `/v3/projects/2a9607b88fb446ec965f0f560c6a73b5` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:39.100352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a9607b88fb446ec965f0f560c6a73b5)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:39.102656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:39.194535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a9607b88fb446ec965f0f560c6a73b5', 'request_id': 'req-034c5ef4-5625-4be9-8610-c9ab7639034c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.207595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a9607b88fb446ec965f0f560c6a73b5', 'request_id': 'req-034c5ef4-5625-4be9-8610-c9ab7639034c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.223947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a9607b88fb446ec965f0f560c6a73b5', 'request_id': 'req-034c5ef4-5625-4be9-8610-c9ab7639034c'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.241649 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] The token cache is being invalidate because project 2a9607b88fb446ec965f0f560c6a73b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:39.242048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.243273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.245663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1336/2681] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:38 2022] DELETE /identity/v3/projects/2a9607b88fb446ec965f0f560c6a73b5 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:39.249644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:39.251440 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:39.267547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d721d9-b688-4e31-81a0-f567bd499ea6 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:39.367049 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:39.368020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:39.368495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:39.368926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] PATH_INFO: `/v3/users/7a2bffdfc69b48c2845ada640acd218b` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:39.370473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a2bffdfc69b48c2845ada640acd218b)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:39.372908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:39.402072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:39.474788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a2bffdfc69b48c2845ada640acd218b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.494220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a2bffdfc69b48c2845ada640acd218b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.495409 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a2bffdfc69b48c2845ada640acd218b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.496192 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a2bffdfc69b48c2845ada640acd218b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.525790 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-003ce9b4-1120-436a-8e28-c92e6ec800a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a2bffdfc69b48c2845ada640acd218b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:39.550205 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1346/2682] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:39 2022] DELETE /identity/v3/users/7a2bffdfc69b48c2845ada640acd218b => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:39.782686 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:39.787147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:39.827204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034c5ef4-5625-4be9-8610-c9ab7639034c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:39.916374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:39.917211 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:39.917953 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:39.918493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] PATH_INFO: `/v3/projects/35427f7112df416a9a69a9f9978b5519` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:39.921138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] RBAC: Authorizing `identity:delete_project(project_id=35427f7112df416a9a69a9f9978b5519)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:39.924885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:39.927708 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:40.001005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35427f7112df416a9a69a9f9978b5519', 'request_id': 'req-e449e171-5c2f-46e8-af88-e364cdbfb593'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.008978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35427f7112df416a9a69a9f9978b5519', 'request_id': 'req-e449e171-5c2f-46e8-af88-e364cdbfb593'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.019134 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35427f7112df416a9a69a9f9978b5519', 'request_id': 'req-e449e171-5c2f-46e8-af88-e364cdbfb593'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.033381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] The token cache is being invalidate because project 35427f7112df416a9a69a9f9978b5519 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:40.033771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.034695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e449e171-5c2f-46e8-af88-e364cdbfb593 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.037060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1337/2683] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:39 2022] DELETE /identity/v3/projects/35427f7112df416a9a69a9f9978b5519 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:40.042641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ae8369-3ab5-44c6-9c70-1f5ae35aed70 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.043250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ae8369-3ab5-44c6-9c70-1f5ae35aed70 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.043250 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ae8369-3ab5-44c6-9c70-1f5ae35aed70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.066610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f6ae8369-3ab5-44c6-9c70-1f5ae35aed70 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:40.068019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f6ae8369-3ab5-44c6-9c70-1f5ae35aed70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:40.180429 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1347/2684] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:40.189463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f111c0b-0ab3-4607-ac87-f25f936c9754 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.190145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f111c0b-0ab3-4607-ac87-f25f936c9754 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.190537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f111c0b-0ab3-4607-ac87-f25f936c9754 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.218352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9f111c0b-0ab3-4607-ac87-f25f936c9754 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:40.219946 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9f111c0b-0ab3-4607-ac87-f25f936c9754 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:40.228540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1338/2685] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:40.233195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ae8369-3ab5-44c6-9c70-1f5ae35aed70 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:40.246843 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ae8369-3ab5-44c6-9c70-1f5ae35aed70 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:40.252608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:40.253344 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.253740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.254164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.255345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:40.255345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:40.257219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:40.273284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1348/2686] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:40.278781 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f111c0b-0ab3-4607-ac87-f25f936c9754 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:40.292485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f111c0b-0ab3-4607-ac87-f25f936c9754 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:40.300718 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:40.301681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.302018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.302538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.303603 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:40.306594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:40.350570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1339/2687] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:40 2022] POST /identity/v3/projects => generated 371 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:40.352080 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:40.363382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b168fce-5208-4a0d-9989-19aaa2dd54af admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:40.481393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ebcf2f-d1bd-4308-9451-7b94183beb5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:40.483009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ebcf2f-d1bd-4308-9451-7b94183beb5f admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.483464 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ebcf2f-d1bd-4308-9451-7b94183beb5f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.483944 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ebcf2f-d1bd-4308-9451-7b94183beb5f admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.487740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ebcf2f-d1bd-4308-9451-7b94183beb5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:40.492836 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ebcf2f-d1bd-4308-9451-7b94183beb5f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:40.515557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:40.522310 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:40.543270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64726e2c-2a45-441e-8057-735a22280968 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:40.550967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:40.551914 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1349/2688] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:40 2022] POST /identity/v3/users => generated 286 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:40.552349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.552967 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.553729 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] PATH_INFO: `/v3/users/7fcc46dae42f46bdbbfd076a6cd23434` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.557922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ebcf2f-d1bd-4308-9451-7b94183beb5f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:40.558239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7fcc46dae42f46bdbbfd076a6cd23434)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:40.563577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:40.580261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ebcf2f-d1bd-4308-9451-7b94183beb5f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:40.589749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d13629f-46f5-4ebb-bc7c-cbe58e57f696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:40.591617 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d13629f-46f5-4ebb-bc7c-cbe58e57f696 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.592380 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d13629f-46f5-4ebb-bc7c-cbe58e57f696 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.593197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d13629f-46f5-4ebb-bc7c-cbe58e57f696 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.593974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d13629f-46f5-4ebb-bc7c-cbe58e57f696 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:40.596085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d13629f-46f5-4ebb-bc7c-cbe58e57f696 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:40.598238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:40.626919 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1350/2689] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:48:40.628413 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d13629f-46f5-4ebb-bc7c-cbe58e57f696 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:40.643005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d13629f-46f5-4ebb-bc7c-cbe58e57f696 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:40.649152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d331379-7e89-46ba-b0a0-e23ec2e0a376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:40.649797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d331379-7e89-46ba-b0a0-e23ec2e0a376 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.650081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d331379-7e89-46ba-b0a0-e23ec2e0a376 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.650366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d331379-7e89-46ba-b0a0-e23ec2e0a376 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.651238 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d331379-7e89-46ba-b0a0-e23ec2e0a376 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:40.652880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d331379-7e89-46ba-b0a0-e23ec2e0a376 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:40.677214 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1351/2690] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:40.684500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d331379-7e89-46ba-b0a0-e23ec2e0a376 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:40.690990 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7fcc46dae42f46bdbbfd076a6cd23434'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.693814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7fcc46dae42f46bdbbfd076a6cd23434'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.704305 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d331379-7e89-46ba-b0a0-e23ec2e0a376 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:40.709513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7fcc46dae42f46bdbbfd076a6cd23434'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.713416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abee3f7-5b6d-41fd-bc04-9e08404e35c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:40.714598 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abee3f7-5b6d-41fd-bc04-9e08404e35c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.715182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abee3f7-5b6d-41fd-bc04-9e08404e35c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.715775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abee3f7-5b6d-41fd-bc04-9e08404e35c5 admin admin] PATH_INFO: `/v3/projects/ebdce8f8e056488cb1e715de7f1f45f9/users/a2b447dc05fd4dd2a5830c026be98f11/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.721466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7fcc46dae42f46bdbbfd076a6cd23434'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.751190 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abee3f7-5b6d-41fd-bc04-9e08404e35c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebdce8f8e056488cb1e715de7f1f45f9, user_id=a2b447dc05fd4dd2a5830c026be98f11, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:40.752334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8a54e1ff-41e4-4f6f-b188-7430abd60f56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7fcc46dae42f46bdbbfd076a6cd23434'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:40.754697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abee3f7-5b6d-41fd-bc04-9e08404e35c5 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:40.771526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1340/2691] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:40 2022] DELETE /identity/v3/users/7fcc46dae42f46bdbbfd076a6cd23434 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:40.778969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb643e2-d331-4a97-a253-d072760ffeb0 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.780257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb643e2-d331-4a97-a253-d072760ffeb0 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.781363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb643e2-d331-4a97-a253-d072760ffeb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.802234 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1352/2692] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:40 2022] PUT /identity/v3/projects/ebdce8f8e056488cb1e715de7f1f45f9/users/a2b447dc05fd4dd2a5830c026be98f11/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:40.824377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d300792-08d1-4341-bd80-086331b4379f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:40.827830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d300792-08d1-4341-bd80-086331b4379f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:40.836096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d300792-08d1-4341-bd80-086331b4379f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:40.840808 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-feb643e2-d331-4a97-a253-d072760ffeb0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:40.844004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-feb643e2-d331-4a97-a253-d072760ffeb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:40.923038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1d300792-08d1-4341-bd80-086331b4379f None None] MFA Rules not processed for user `a2b447dc05fd4dd2a5830c026be98f11`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:40.925829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1d300792-08d1-4341-bd80-086331b4379f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:40.939450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:41.000543 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1341/2693] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:41.014855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb643e2-d331-4a97-a253-d072760ffeb0 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:41.016331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-feb643e2-d331-4a97-a253-d072760ffeb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:41.068061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1353/2694] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:41.081061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d300792-08d1-4341-bd80-086331b4379f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:41.082562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1d300792-08d1-4341-bd80-086331b4379f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:41.121731 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb643e2-d331-4a97-a253-d072760ffeb0 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:41.128589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:41.129863 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:41.130137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:41.130539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:41.142519 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:41.144184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:41.159941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1342/2695] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:41.171812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d300792-08d1-4341-bd80-086331b4379f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:41.176414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee64528-2d5a-4d13-84c3-ea9495524831 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:41.177504 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee64528-2d5a-4d13-84c3-ea9495524831 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:41.178166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee64528-2d5a-4d13-84c3-ea9495524831 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:41.178763 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee64528-2d5a-4d13-84c3-ea9495524831 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:41.193330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee64528-2d5a-4d13-84c3-ea9495524831 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:41.195285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee64528-2d5a-4d13-84c3-ea9495524831 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:41.207745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1354/2696] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:41 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:41.239824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:41.242463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:41.260150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f2503a-8c6a-4468-8b8f-105b91f8f6f1 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:41.266788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:41.267635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:41.268040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:41.268420 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] PATH_INFO: `/v3/users/a2b447dc05fd4dd2a5830c026be98f11` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:41.269841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2b447dc05fd4dd2a5830c026be98f11)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:41.273021 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:41.348138 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2b447dc05fd4dd2a5830c026be98f11'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.349853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2b447dc05fd4dd2a5830c026be98f11'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.374333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2b447dc05fd4dd2a5830c026be98f11'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.388752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2b447dc05fd4dd2a5830c026be98f11'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.439440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2b447dc05fd4dd2a5830c026be98f11'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.461876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1343/2697] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:41 2022] DELETE /identity/v3/users/a2b447dc05fd4dd2a5830c026be98f11 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:41.468976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fa30f2-dfd3-429c-860a-e1c5fb391f92 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:41.469803 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fa30f2-dfd3-429c-860a-e1c5fb391f92 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:41.470424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fa30f2-dfd3-429c-860a-e1c5fb391f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:41.497516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-83fa30f2-dfd3-429c-860a-e1c5fb391f92 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:41.499180 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-83fa30f2-dfd3-429c-860a-e1c5fb391f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:41.611514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:41.633064 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:41.634713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1355/2698] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:41.635481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:41.653928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fa30f2-dfd3-429c-860a-e1c5fb391f92 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:41.656938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-83fa30f2-dfd3-429c-860a-e1c5fb391f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:41.665218 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e102b3b3-b444-4ec1-82fa-277c75d18685 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:41.685060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:41.686154 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:41.686782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:41.688023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] PATH_INFO: `/v3/projects/f37dab88a82543b78761778a8e6e6834` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:41.691227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=f37dab88a82543b78761778a8e6e6834)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:41.696915 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:41.757630 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fa30f2-dfd3-429c-860a-e1c5fb391f92 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:41.762541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb386be-47aa-4ff8-90c9-a2eb3cf349fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:41.763275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb386be-47aa-4ff8-90c9-a2eb3cf349fe service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:41.764169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb386be-47aa-4ff8-90c9-a2eb3cf349fe service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:41.764608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb386be-47aa-4ff8-90c9-a2eb3cf349fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:41.777760 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb386be-47aa-4ff8-90c9-a2eb3cf349fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:41.780761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb386be-47aa-4ff8-90c9-a2eb3cf349fe service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:41.794867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1356/2699] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:41.802314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:41.806501 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f37dab88a82543b78761778a8e6e6834', 'request_id': 'req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.827801 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f37dab88a82543b78761778a8e6e6834', 'request_id': 'req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.841006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f37dab88a82543b78761778a8e6e6834', 'request_id': 'req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.873382 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] The token cache is being invalidate because project f37dab88a82543b78761778a8e6e6834 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:41.873989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.875567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:41.878485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1344/2700] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:41 2022] DELETE /identity/v3/projects/f37dab88a82543b78761778a8e6e6834 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:41.885240 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb386be-47aa-4ff8-90c9-a2eb3cf349fe service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:41.914819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb386be-47aa-4ff8-90c9-a2eb3cf349fe service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:42.025893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26018004-2513-4843-aabd-06b776324ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:42.026878 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26018004-2513-4843-aabd-06b776324ced admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:42.027505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26018004-2513-4843-aabd-06b776324ced admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:42.027721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26018004-2513-4843-aabd-06b776324ced admin admin] PATH_INFO: `/v3/users/0aebb188b0f24c6293c3b2dea8ec8382` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:42.029498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26018004-2513-4843-aabd-06b776324ced admin admin] RBAC: Authorizing `identity:delete_user(user_id=0aebb188b0f24c6293c3b2dea8ec8382)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:42.032217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26018004-2513-4843-aabd-06b776324ced admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:42.100588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-26018004-2513-4843-aabd-06b776324ced admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0aebb188b0f24c6293c3b2dea8ec8382'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.108963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-26018004-2513-4843-aabd-06b776324ced admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0aebb188b0f24c6293c3b2dea8ec8382'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.110274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-26018004-2513-4843-aabd-06b776324ced admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0aebb188b0f24c6293c3b2dea8ec8382'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.111036 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-26018004-2513-4843-aabd-06b776324ced admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0aebb188b0f24c6293c3b2dea8ec8382'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.120599 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-26018004-2513-4843-aabd-06b776324ced admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0aebb188b0f24c6293c3b2dea8ec8382'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.156837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1357/2701] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:41 2022] DELETE /identity/v3/users/0aebb188b0f24c6293c3b2dea8ec8382 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:42.487919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:42.488368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:42.528627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c35591e-f809-4a6c-ab9e-cafeb4a636ef admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:42.684424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26018004-2513-4843-aabd-06b776324ced admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:42.686714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-26018004-2513-4843-aabd-06b776324ced admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:42.693493 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:42.694336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:42.695786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:42.696197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] PATH_INFO: `/v3/projects/ebdce8f8e056488cb1e715de7f1f45f9` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:42.703051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ebdce8f8e056488cb1e715de7f1f45f9)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:42.704993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:42.714597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26018004-2513-4843-aabd-06b776324ced admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:42.727510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:42.729177 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:42.729667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:42.730276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] PATH_INFO: `/v3/users/b191436341a9487d9d9b80f04835522a` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:42.733163 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b191436341a9487d9d9b80f04835522a)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:42.744528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:42.768444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:42.846767 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b191436341a9487d9d9b80f04835522a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.857004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ebdce8f8e056488cb1e715de7f1f45f9', 'request_id': 'req-6e8ae08c-b541-452f-908c-e61e04bc4c1f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.859498 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b191436341a9487d9d9b80f04835522a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.860960 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b191436341a9487d9d9b80f04835522a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.861739 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b191436341a9487d9d9b80f04835522a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.873663 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ebdce8f8e056488cb1e715de7f1f45f9', 'request_id': 'req-6e8ae08c-b541-452f-908c-e61e04bc4c1f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.879531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-e8930d2e-2e32-4fee-afc8-aa19c1f4ed6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b191436341a9487d9d9b80f04835522a'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.887449 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ebdce8f8e056488cb1e715de7f1f45f9', 'request_id': 'req-6e8ae08c-b541-452f-908c-e61e04bc4c1f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.903248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] The token cache is being invalidate because project ebdce8f8e056488cb1e715de7f1f45f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:42.903615 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.904524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:42.909376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1345/2702] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:42 2022] DELETE /identity/v3/projects/ebdce8f8e056488cb1e715de7f1f45f9 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:42.909376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:42.910361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:42.924041 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1358/2703] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:42 2022] DELETE /identity/v3/users/b191436341a9487d9d9b80f04835522a => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:42.925928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b9ea9-75a0-47b5-b683-97609d4cd317 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:42.926455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b9ea9-75a0-47b5-b683-97609d4cd317 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:42.927218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b9ea9-75a0-47b5-b683-97609d4cd317 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:42.976423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-865b9ea9-75a0-47b5-b683-97609d4cd317 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:42.977888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-865b9ea9-75a0-47b5-b683-97609d4cd317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.032158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ae08c-b541-452f-908c-e61e04bc4c1f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:43.039558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:43.109669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1359/2704] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:43.119555 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9345201-1b37-43af-ab81-6bcc63524c2e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:43.119958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9345201-1b37-43af-ab81-6bcc63524c2e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:43.120642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9345201-1b37-43af-ab81-6bcc63524c2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:43.160029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-b9345201-1b37-43af-ab81-6bcc63524c2e None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:43.161945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b9345201-1b37-43af-ab81-6bcc63524c2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.165810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:43.166597 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:43.167002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:43.167270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] PATH_INFO: `/v3/projects/baeec031611e45febe34e52feb2fa49f` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:43.169419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=baeec031611e45febe34e52feb2fa49f)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:43.172850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:43.175095 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1360/2705] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:43.175968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e78ab-a7f3-46e5-976b-95b1198e8525 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:43.176431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e78ab-a7f3-46e5-976b-95b1198e8525 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:43.177082 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e78ab-a7f3-46e5-976b-95b1198e8525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:43.215402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-db1e78ab-a7f3-46e5-976b-95b1198e8525 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:43.218365 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-db1e78ab-a7f3-46e5-976b-95b1198e8525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.231035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1361/2706] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:43.231395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e78ab-a7f3-46e5-976b-95b1198e8525 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:43.243017 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-db1e78ab-a7f3-46e5-976b-95b1198e8525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.258206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'baeec031611e45febe34e52feb2fa49f', 'request_id': 'req-29d18dae-d9c8-4298-85bc-276fdf0125d8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.271918 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'baeec031611e45febe34e52feb2fa49f', 'request_id': 'req-29d18dae-d9c8-4298-85bc-276fdf0125d8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.285304 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'baeec031611e45febe34e52feb2fa49f', 'request_id': 'req-29d18dae-d9c8-4298-85bc-276fdf0125d8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.304000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] The token cache is being invalidate because project baeec031611e45febe34e52feb2fa49f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:43.304251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.305758 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.308308 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1346/2707] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:42 2022] DELETE /identity/v3/projects/baeec031611e45febe34e52feb2fa49f => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:43.309461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:43.314486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.375935 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e78ab-a7f3-46e5-976b-95b1198e8525 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:43.439803 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:43.448440 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d18dae-d9c8-4298-85bc-276fdf0125d8 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:43.456270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:43.456875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:43.457254 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:43.457947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:43.458333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:43.459251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:43.460767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:43.479668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1347/2708] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:43.485725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:43.485725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.492684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:43.493319 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:43.493765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:43.493982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:43.495829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.500024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0d6683-df81-4dab-92af-38431af9c16c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:43.508390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:43.509752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:43.510050 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:43.510318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] PATH_INFO: `/v3/users/f4b42133fdf74666ac54d8a4e2a8cd90` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:43.511902 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4b42133fdf74666ac54d8a4e2a8cd90)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:43.514448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:43.515788 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:43.518693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:43.535210 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1362/2709] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:43.536462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:43.538377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.556700 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d58025-f9e7-419e-b703-30492f9b4a52 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:43.559979 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:43.566817 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:43.570002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:43.570314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:43.571035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] PATH_INFO: `/v3/users/003dd6612cda48618966db8c0788b794` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:43.573969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=003dd6612cda48618966db8c0788b794)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:43.577807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:43.596722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4b42133fdf74666ac54d8a4e2a8cd90'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.597837 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4b42133fdf74666ac54d8a4e2a8cd90'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.619809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4b42133fdf74666ac54d8a4e2a8cd90'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.647119 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4b42133fdf74666ac54d8a4e2a8cd90'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.673172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4b42133fdf74666ac54d8a4e2a8cd90'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.679359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '003dd6612cda48618966db8c0788b794'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.688666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1348/2710] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:43 2022] DELETE /identity/v3/users/f4b42133fdf74666ac54d8a4e2a8cd90 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:43.688666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:43.688666 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:43.713829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '003dd6612cda48618966db8c0788b794'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.715063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '003dd6612cda48618966db8c0788b794'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.716510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '003dd6612cda48618966db8c0788b794'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.743683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '003dd6612cda48618966db8c0788b794'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:43.807583 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1363/2711] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:48:43 2022] DELETE /identity/v3/users/003dd6612cda48618966db8c0788b794 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:43.863916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a8ec7-6134-4ab4-9d76-f2598cd99ffd admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:43.979131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:43.979969 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:43.980442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:43.980834 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:43.981908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:43.984782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.042872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1349/2712] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:43 2022] POST /identity/v3/projects => generated 385 bytes in 357 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:44.049531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.051475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.067802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe989ad-7752-461e-8b70-5a4e412cfd31 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:44.076038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.078804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.170710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:44.172939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.173668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.174352 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.179255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:44.181568 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:44.184313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.225550 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff37fc7-c8f4-43ce-92ca-a30f6b96a90c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:44.231281 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:44.231930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.232204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.233004 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] PATH_INFO: `/v3/projects/0084e9c31be24b248672a312e039b68c` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.234165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0084e9c31be24b248672a312e039b68c)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:44.236180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.250723 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1364/2713] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:44 2022] POST /identity/v3/users => generated 293 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:44.252782 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.257680 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.284040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c09e058-b54b-4828-bd00-a6e37723d2cb admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:44.292048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:44.293223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.293749 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.295218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] PATH_INFO: `/v3/projects/05e6d1afb63446af887a1de133b7bf2e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.296445 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=05e6d1afb63446af887a1de133b7bf2e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:44.302127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.338612 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0084e9c31be24b248672a312e039b68c', 'request_id': 'req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.353028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=84,keystone:DELETE=12,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:44.354317 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0084e9c31be24b248672a312e039b68c', 'request_id': 'req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.368097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0084e9c31be24b248672a312e039b68c', 'request_id': 'req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.374165 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05e6d1afb63446af887a1de133b7bf2e', 'request_id': 'req-fed6108d-29a0-4d26-9893-3c588ba762f8'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.382607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] The token cache is being invalidate because project 0084e9c31be24b248672a312e039b68c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:44.383172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.385016 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.386552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1350/2714] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:44 2022] DELETE /identity/v3/projects/0084e9c31be24b248672a312e039b68c => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:48:44.386552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.387949 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.388151 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05e6d1afb63446af887a1de133b7bf2e', 'request_id': 'req-fed6108d-29a0-4d26-9893-3c588ba762f8'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.398334 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05e6d1afb63446af887a1de133b7bf2e', 'request_id': 'req-fed6108d-29a0-4d26-9893-3c588ba762f8'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.414571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] The token cache is being invalidate because project 05e6d1afb63446af887a1de133b7bf2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:44.415031 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.415853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.417765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1365/2715] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:44 2022] DELETE /identity/v3/projects/05e6d1afb63446af887a1de133b7bf2e => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:44.419257 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.421522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.506180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc6c5cf-2344-4bf2-b16f-7be9c35dcc96 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:44.529569 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed6108d-29a0-4d26-9893-3c588ba762f8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:44.537665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:44.538774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.539273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.539743 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] PATH_INFO: `/v3/projects/b13e875bc4684a0ba7b5466dfc88deb5` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.541718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=b13e875bc4684a0ba7b5466dfc88deb5)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:44.545424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.575692 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd89c8d-0ac8-4061-b7ec-837717a4c640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:44.577275 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd89c8d-0ac8-4061-b7ec-837717a4c640 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.577716 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd89c8d-0ac8-4061-b7ec-837717a4c640 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.578123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd89c8d-0ac8-4061-b7ec-837717a4c640 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.579052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd89c8d-0ac8-4061-b7ec-837717a4c640 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:44.580974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd89c8d-0ac8-4061-b7ec-837717a4c640 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.593690 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1351/2716] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:44 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:44.595761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5161f3cf-2381-4bab-8d9e-1db2b3742dff None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.596241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5161f3cf-2381-4bab-8d9e-1db2b3742dff None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.596670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5161f3cf-2381-4bab-8d9e-1db2b3742dff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.620978 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5161f3cf-2381-4bab-8d9e-1db2b3742dff None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:44.622526 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5161f3cf-2381-4bab-8d9e-1db2b3742dff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.629957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:44.632401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b13e875bc4684a0ba7b5466dfc88deb5', 'request_id': 'req-8e570f10-5654-425f-975c-c405331344ab'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.643219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b13e875bc4684a0ba7b5466dfc88deb5', 'request_id': 'req-8e570f10-5654-425f-975c-c405331344ab'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.653028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b13e875bc4684a0ba7b5466dfc88deb5', 'request_id': 'req-8e570f10-5654-425f-975c-c405331344ab'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.677083 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] The token cache is being invalidate because project b13e875bc4684a0ba7b5466dfc88deb5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:48:44.677633 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.678525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-8e570f10-5654-425f-975c-c405331344ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:48:44.680368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1366/2717] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:48:44 2022] DELETE /identity/v3/projects/b13e875bc4684a0ba7b5466dfc88deb5 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:44.681685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf7d97-040c-4033-bf27-4a10d7378233 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.681972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf7d97-040c-4033-bf27-4a10d7378233 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.682251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf7d97-040c-4033-bf27-4a10d7378233 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.706417 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-ecdf7d97-040c-4033-bf27-4a10d7378233 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:44.708608 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ecdf7d97-040c-4033-bf27-4a10d7378233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.715490 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1352/2718] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:44.716357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5161f3cf-2381-4bab-8d9e-1db2b3742dff None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.717873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5161f3cf-2381-4bab-8d9e-1db2b3742dff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.730773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1367/2719] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:44.730773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f7264-57de-4f1f-a678-51ea32753021 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.730773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f7264-57de-4f1f-a678-51ea32753021 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.730773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6f7264-57de-4f1f-a678-51ea32753021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.731875 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5161f3cf-2381-4bab-8d9e-1db2b3742dff None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:44.737503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f0707e-c89d-4031-8eb5-2b7d9963aeb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:44.738116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0707e-c89d-4031-8eb5-2b7d9963aeb8 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.738392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0707e-c89d-4031-8eb5-2b7d9963aeb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.738658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0707e-c89d-4031-8eb5-2b7d9963aeb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.739573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f0707e-c89d-4031-8eb5-2b7d9963aeb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:44.741269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f0707e-c89d-4031-8eb5-2b7d9963aeb8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.752733 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1353/2720] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:44 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:44.753917 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab043c42-8e20-4fcb-bbd0-e6585d6037d1 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.754181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab043c42-8e20-4fcb-bbd0-e6585d6037d1 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.754458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab043c42-8e20-4fcb-bbd0-e6585d6037d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.761912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-dd6f7264-57de-4f1f-a678-51ea32753021 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:44.764562 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dd6f7264-57de-4f1f-a678-51ea32753021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.778467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1368/2721] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:44.791431 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-ab043c42-8e20-4fcb-bbd0-e6585d6037d1 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:44.792695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ab043c42-8e20-4fcb-bbd0-e6585d6037d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.792893 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05025412-1318-4956-a488-54580a788f7a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.793321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05025412-1318-4956-a488-54580a788f7a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.793746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05025412-1318-4956-a488-54580a788f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.801182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1354/2722] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:44.802026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab043c42-8e20-4fcb-bbd0-e6585d6037d1 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.812780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab043c42-8e20-4fcb-bbd0-e6585d6037d1 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:44.821730 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb105af9-d652-4ca1-9815-7bef60feff35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:44.823057 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb105af9-d652-4ca1-9815-7bef60feff35 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.823546 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb105af9-d652-4ca1-9815-7bef60feff35 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.824014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb105af9-d652-4ca1-9815-7bef60feff35 admin admin] PATH_INFO: `/v3/projects/c814b0ae26d84c08b9c27e1f11d3b2ff/users/9f7d298884b743f49760437628fc65aa/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.827384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-05025412-1318-4956-a488-54580a788f7a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:44.828637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-05025412-1318-4956-a488-54580a788f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.839201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1369/2723] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:44.841547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d987b-834b-4ff8-87ea-bbc0b8fbbe18 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.843309 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d987b-834b-4ff8-87ea-bbc0b8fbbe18 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.843861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d987b-834b-4ff8-87ea-bbc0b8fbbe18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.854238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb105af9-d652-4ca1-9815-7bef60feff35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c814b0ae26d84c08b9c27e1f11d3b2ff, user_id=9f7d298884b743f49760437628fc65aa, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:44.859831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb105af9-d652-4ca1-9815-7bef60feff35 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.865861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-327d987b-834b-4ff8-87ea-bbc0b8fbbe18 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:44.867266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-327d987b-834b-4ff8-87ea-bbc0b8fbbe18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:44.877221 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1370/2724] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:44.878169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327d987b-834b-4ff8-87ea-bbc0b8fbbe18 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.886495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1355/2725] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:44 2022] PUT /identity/v3/projects/c814b0ae26d84c08b9c27e1f11d3b2ff/users/9f7d298884b743f49760437628fc65aa/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:44.888297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb105af9-d652-4ca1-9815-7bef60feff35 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.892745 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327d987b-834b-4ff8-87ea-bbc0b8fbbe18 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:44.919646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:44.968666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:44.969286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:44.969563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:44.969830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:44.970485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:44.970826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:44.972435 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:44.986162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1371/2726] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:44.987054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:44.998189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09e7c5-adb1-4485-ac74-c6224ea249ee admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.003233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a3251f-0da7-411a-a1c4-ce79025104aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.003844 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a3251f-0da7-411a-a1c4-ce79025104aa admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.004120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a3251f-0da7-411a-a1c4-ce79025104aa admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.004424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a3251f-0da7-411a-a1c4-ce79025104aa admin admin] PATH_INFO: `/v3/domains` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.005091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a3251f-0da7-411a-a1c4-ce79025104aa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103002) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:45.005400 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a3251f-0da7-411a-a1c4-ce79025104aa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.007079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a3251f-0da7-411a-a1c4-ce79025104aa admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.016116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb105af9-d652-4ca1-9815-7bef60feff35 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.021368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1372/2727] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:45.022628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180facac-677a-4386-bd6e-08375d9d5990 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.022943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180facac-677a-4386-bd6e-08375d9d5990 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.023224 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180facac-677a-4386-bd6e-08375d9d5990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.027017 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.027997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.028445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.028888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.030061 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:48:45.030616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.033694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.043339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-180facac-677a-4386-bd6e-08375d9d5990 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:45.044571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-180facac-677a-4386-bd6e-08375d9d5990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:45.053147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1373/2728] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:45.053970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180facac-677a-4386-bd6e-08375d9d5990 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.059038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1356/2729] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:48:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:48:45.059038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.064263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180facac-677a-4386-bd6e-08375d9d5990 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.069302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc4465b-68bb-4d9f-bc42-47bedb6366ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.069922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4465b-68bb-4d9f-bc42-47bedb6366ae admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.070204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4465b-68bb-4d9f-bc42-47bedb6366ae admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.071252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4465b-68bb-4d9f-bc42-47bedb6366ae admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.071252 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc4465b-68bb-4d9f-bc42-47bedb6366ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.073069 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc4465b-68bb-4d9f-bc42-47bedb6366ae admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.078131 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38a7439-4677-45b1-8e61-a6e3bd79725f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.086527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.087572 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.088043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.088510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.089658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.094595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.107304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1374/2730] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:45 2022] POST /identity/v3/projects => generated 379 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:45.109311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc4465b-68bb-4d9f-bc42-47bedb6366ae admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.154767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1357/2731] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:45 2022] POST /identity/v3/projects => generated 371 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:45.156365 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.163982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:45.213665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc4465b-68bb-4d9f-bc42-47bedb6366ae admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.221774 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:45.239194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a65927-f27a-44ec-be54-16859dbe9920 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.243183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d7222-35b8-41e6-a006-3ce0f92d2c9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.243800 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d7222-35b8-41e6-a006-3ce0f92d2c9b service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.244069 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d7222-35b8-41e6-a006-3ce0f92d2c9b service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.244378 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d7222-35b8-41e6-a006-3ce0f92d2c9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.302509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5c25ad-acb5-45dc-be50-23669f5db502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.303388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5c25ad-acb5-45dc-be50-23669f5db502 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.304366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5c25ad-acb5-45dc-be50-23669f5db502 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.304816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5c25ad-acb5-45dc-be50-23669f5db502 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.306604 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5c25ad-acb5-45dc-be50-23669f5db502 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.311091 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5c25ad-acb5-45dc-be50-23669f5db502 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.329342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d7222-35b8-41e6-a006-3ce0f92d2c9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.331144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d7222-35b8-41e6-a006-3ce0f92d2c9b service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.345453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1358/2732] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:45.346998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d7222-35b8-41e6-a006-3ce0f92d2c9b service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.358713 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1375/2733] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:45 2022] POST /identity/v3/projects => generated 365 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:45.359671 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5c25ad-acb5-45dc-be50-23669f5db502 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.367027 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d7222-35b8-41e6-a006-3ce0f92d2c9b service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.369341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:45.466044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5c25ad-acb5-45dc-be50-23669f5db502 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.473616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba5e9df-b857-47b7-9264-b4bc3a2bbb7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.474805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba5e9df-b857-47b7-9264-b4bc3a2bbb7e admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.475699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba5e9df-b857-47b7-9264-b4bc3a2bbb7e admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.475835 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba5e9df-b857-47b7-9264-b4bc3a2bbb7e admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.476943 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba5e9df-b857-47b7-9264-b4bc3a2bbb7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.480005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba5e9df-b857-47b7-9264-b4bc3a2bbb7e admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.484042 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0757e3b2-15a1-4330-a760-203cba856412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.484867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0757e3b2-15a1-4330-a760-203cba856412 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.485148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0757e3b2-15a1-4330-a760-203cba856412 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.485588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0757e3b2-15a1-4330-a760-203cba856412 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.486252 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0757e3b2-15a1-4330-a760-203cba856412 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.488286 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0757e3b2-15a1-4330-a760-203cba856412 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.518793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1376/2734] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:45 2022] POST /identity/v3/users => generated 286 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:45.518793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba5e9df-b857-47b7-9264-b4bc3a2bbb7e admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.520645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1359/2735] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:45 2022] POST /identity/v3/users => generated 290 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:45.521559 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0757e3b2-15a1-4330-a760-203cba856412 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.535179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba5e9df-b857-47b7-9264-b4bc3a2bbb7e admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.535570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0757e3b2-15a1-4330-a760-203cba856412 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.541086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470668c8-05c5-4b30-9cc4-a8a98fe8a296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.541876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470668c8-05c5-4b30-9cc4-a8a98fe8a296 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.542714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470668c8-05c5-4b30-9cc4-a8a98fe8a296 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.542714 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470668c8-05c5-4b30-9cc4-a8a98fe8a296 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.543595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470668c8-05c5-4b30-9cc4-a8a98fe8a296 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.544000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7639f2-576c-48e2-a7bf-e9acb0e12e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.545663 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7639f2-576c-48e2-a7bf-e9acb0e12e77 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.546054 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470668c8-05c5-4b30-9cc4-a8a98fe8a296 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.546449 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7639f2-576c-48e2-a7bf-e9acb0e12e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.547251 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7639f2-576c-48e2-a7bf-e9acb0e12e77 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.547882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7639f2-576c-48e2-a7bf-e9acb0e12e77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.551444 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7639f2-576c-48e2-a7bf-e9acb0e12e77 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.565664 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1360/2736] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:45.568349 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470668c8-05c5-4b30-9cc4-a8a98fe8a296 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.583908 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1377/2737] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:45 2022] POST /identity/v3/users => generated 283 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:45.585397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7639f2-576c-48e2-a7bf-e9acb0e12e77 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.587165 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470668c8-05c5-4b30-9cc4-a8a98fe8a296 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.593783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6e39f1-b409-4edd-89e3-6893a0e52be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.594413 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6e39f1-b409-4edd-89e3-6893a0e52be7 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.595478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6e39f1-b409-4edd-89e3-6893a0e52be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.595478 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6e39f1-b409-4edd-89e3-6893a0e52be7 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.595815 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6e39f1-b409-4edd-89e3-6893a0e52be7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.597844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6e39f1-b409-4edd-89e3-6893a0e52be7 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.602815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7639f2-576c-48e2-a7bf-e9acb0e12e77 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.610076 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7c6896-e755-4418-a501-5274fc137bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.611053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7c6896-e755-4418-a501-5274fc137bb3 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.611951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7c6896-e755-4418-a501-5274fc137bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.611951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7c6896-e755-4418-a501-5274fc137bb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.613377 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7c6896-e755-4418-a501-5274fc137bb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.615708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7c6896-e755-4418-a501-5274fc137bb3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.622985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1361/2738] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:45.622985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6e39f1-b409-4edd-89e3-6893a0e52be7 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.642809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1378/2739] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:45.642809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7c6896-e755-4418-a501-5274fc137bb3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.645130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6e39f1-b409-4edd-89e3-6893a0e52be7 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.653710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7c6896-e755-4418-a501-5274fc137bb3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.655998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8115d0f-1471-4a51-bbea-470239d6f60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.656974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8115d0f-1471-4a51-bbea-470239d6f60f admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.657258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8115d0f-1471-4a51-bbea-470239d6f60f admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.657525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8115d0f-1471-4a51-bbea-470239d6f60f admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.658337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8115d0f-1471-4a51-bbea-470239d6f60f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.660068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8115d0f-1471-4a51-bbea-470239d6f60f admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.663463 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24496e07-7665-4e72-94ea-f08bac86a6ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.664462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24496e07-7665-4e72-94ea-f08bac86a6ff admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.664811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24496e07-7665-4e72-94ea-f08bac86a6ff admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.665162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24496e07-7665-4e72-94ea-f08bac86a6ff admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.666158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24496e07-7665-4e72-94ea-f08bac86a6ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.668509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24496e07-7665-4e72-94ea-f08bac86a6ff admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.672856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1362/2740] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:45.674194 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8115d0f-1471-4a51-bbea-470239d6f60f admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.691835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8115d0f-1471-4a51-bbea-470239d6f60f admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.695004 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1379/2741] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:45.697276 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24496e07-7665-4e72-94ea-f08bac86a6ff admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.700659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29e9832-16c5-4f0b-90ee-11753cdf54da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.702300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29e9832-16c5-4f0b-90ee-11753cdf54da admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.702841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29e9832-16c5-4f0b-90ee-11753cdf54da admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.703579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29e9832-16c5-4f0b-90ee-11753cdf54da admin admin] PATH_INFO: `/v3/projects/b1f240a0b08841edb5906875c5dcf3e7/users/17a5395280b342de9a39f2a984a2c0ba/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.715302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24496e07-7665-4e72-94ea-f08bac86a6ff admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.723316 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8a110a-ac2f-4739-82e1-6d982c40232b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.724396 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8a110a-ac2f-4739-82e1-6d982c40232b admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.724838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8a110a-ac2f-4739-82e1-6d982c40232b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.725274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8a110a-ac2f-4739-82e1-6d982c40232b admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.726094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8a110a-ac2f-4739-82e1-6d982c40232b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.728181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8a110a-ac2f-4739-82e1-6d982c40232b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.733403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29e9832-16c5-4f0b-90ee-11753cdf54da admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1f240a0b08841edb5906875c5dcf3e7, user_id=17a5395280b342de9a39f2a984a2c0ba, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.737771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29e9832-16c5-4f0b-90ee-11753cdf54da admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.748213 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1380/2742] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:45.754351 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8a110a-ac2f-4739-82e1-6d982c40232b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.760752 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1363/2743] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:45 2022] PUT /identity/v3/projects/b1f240a0b08841edb5906875c5dcf3e7/users/17a5395280b342de9a39f2a984a2c0ba/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:45.762926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29e9832-16c5-4f0b-90ee-11753cdf54da admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:45.779924 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8a110a-ac2f-4739-82e1-6d982c40232b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.815373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:45.915926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29e9832-16c5-4f0b-90ee-11753cdf54da admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:45.926919 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7a4bf1-af46-40e8-9cb3-d15b51ecc83a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.928123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7a4bf1-af46-40e8-9cb3-d15b51ecc83a admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.928582 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7a4bf1-af46-40e8-9cb3-d15b51ecc83a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.929038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7a4bf1-af46-40e8-9cb3-d15b51ecc83a admin admin] PATH_INFO: `/v3/projects/fb29932410eb4588868635137f669d23/users/953f4b84484541e4a33365cd01788be0/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.933168 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818500a2-b12a-439c-ba0f-1879d12f0366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:45.934462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818500a2-b12a-439c-ba0f-1879d12f0366 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.935073 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818500a2-b12a-439c-ba0f-1879d12f0366 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.936280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818500a2-b12a-439c-ba0f-1879d12f0366 admin admin] PATH_INFO: `/v3/projects/36e090f60dd244cea327de965ad9af13/users/1addff9d7c7348de827c4eba89d41660/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:45.956074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7a4bf1-af46-40e8-9cb3-d15b51ecc83a admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb29932410eb4588868635137f669d23, user_id=953f4b84484541e4a33365cd01788be0, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.959895 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7a4bf1-af46-40e8-9cb3-d15b51ecc83a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.969980 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818500a2-b12a-439c-ba0f-1879d12f0366 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36e090f60dd244cea327de965ad9af13, user_id=1addff9d7c7348de827c4eba89d41660, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:45.976686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818500a2-b12a-439c-ba0f-1879d12f0366 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:45.986683 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1364/2744] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:45 2022] PUT /identity/v3/projects/fb29932410eb4588868635137f669d23/users/953f4b84484541e4a33365cd01788be0/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:45.991516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375964b-de21-4735-b9af-f4061f6aca3f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:45.992273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375964b-de21-4735-b9af-f4061f6aca3f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:45.993950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375964b-de21-4735-b9af-f4061f6aca3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.019132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1381/2745] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:45 2022] PUT /identity/v3/projects/36e090f60dd244cea327de965ad9af13/users/1addff9d7c7348de827c4eba89d41660/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:46.027189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3948cf47-0dab-4245-9570-0deea5222a97 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:46.027655 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3948cf47-0dab-4245-9570-0deea5222a97 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:46.028138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3948cf47-0dab-4245-9570-0deea5222a97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.030783 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-5375964b-de21-4735-b9af-f4061f6aca3f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:46.032726 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5375964b-de21-4735-b9af-f4061f6aca3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:46.083515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3948cf47-0dab-4245-9570-0deea5222a97 None None] MFA Rules not processed for user `953f4b84484541e4a33365cd01788be0`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:46.085359 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3948cf47-0dab-4245-9570-0deea5222a97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:46.115199 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:46.145315 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1365/2746] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:46.146852 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed6f099-5197-4e4d-8e8a-5c150f00f225 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:46.147167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed6f099-5197-4e4d-8e8a-5c150f00f225 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:46.147575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed6f099-5197-4e4d-8e8a-5c150f00f225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.174374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2ed6f099-5197-4e4d-8e8a-5c150f00f225 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:46.175905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2ed6f099-5197-4e4d-8e8a-5c150f00f225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:46.186156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1366/2747] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:46.188361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed6f099-5197-4e4d-8e8a-5c150f00f225 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:46.197829 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1382/2748] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:46 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:46.204873 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3948cf47-0dab-4245-9570-0deea5222a97 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:46.302993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed6f099-5197-4e4d-8e8a-5c150f00f225 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:46.311047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f98a0-738c-438b-9169-e1bf84995686 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:46.312152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f98a0-738c-438b-9169-e1bf84995686 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:46.312870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f98a0-738c-438b-9169-e1bf84995686 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:46.313352 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f98a0-738c-438b-9169-e1bf84995686 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.314770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3948cf47-0dab-4245-9570-0deea5222a97 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:46.334670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2f98a0-738c-438b-9169-e1bf84995686 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:46.336041 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2f98a0-738c-438b-9169-e1bf84995686 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:46.340354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2536485-9acb-48cd-bf92-8c6f1952eaa7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:46.341063 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2536485-9acb-48cd-bf92-8c6f1952eaa7 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:46.341388 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2536485-9acb-48cd-bf92-8c6f1952eaa7 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:46.342233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2536485-9acb-48cd-bf92-8c6f1952eaa7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.354814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1367/2749] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:46.356213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f98a0-738c-438b-9169-e1bf84995686 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:46.372288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2536485-9acb-48cd-bf92-8c6f1952eaa7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:46.373089 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f98a0-738c-438b-9169-e1bf84995686 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:46.376653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2536485-9acb-48cd-bf92-8c6f1952eaa7 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:46.385700 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18abb2c-fc77-4549-a244-db3dfa476631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:46.387876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18abb2c-fc77-4549-a244-db3dfa476631 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:46.388943 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18abb2c-fc77-4549-a244-db3dfa476631 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:46.389424 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18abb2c-fc77-4549-a244-db3dfa476631 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.390856 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18abb2c-fc77-4549-a244-db3dfa476631 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:46.391001 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1383/2750] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:46.394143 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18abb2c-fc77-4549-a244-db3dfa476631 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:46.441514 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1368/2751] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:46 2022] POST /identity/v3/projects => generated 365 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:46.447576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2536485-9acb-48cd-bf92-8c6f1952eaa7 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:46.462241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2536485-9acb-48cd-bf92-8c6f1952eaa7 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:46.578990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5562db3b-6c63-4b98-a19c-b61ec8fb90ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:46.580894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5562db3b-6c63-4b98-a19c-b61ec8fb90ad admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:46.581526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5562db3b-6c63-4b98-a19c-b61ec8fb90ad admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:46.582038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5562db3b-6c63-4b98-a19c-b61ec8fb90ad admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.583328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5562db3b-6c63-4b98-a19c-b61ec8fb90ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:46.586509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5562db3b-6c63-4b98-a19c-b61ec8fb90ad admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:46.636382 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1384/2752] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:46 2022] POST /identity/v3/users => generated 289 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:46.638437 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18abb2c-fc77-4549-a244-db3dfa476631 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:46.678660 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18abb2c-fc77-4549-a244-db3dfa476631 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:46.686040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d85bd6d-c0d6-44ec-9250-69faca9b28d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:46.687742 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d85bd6d-c0d6-44ec-9250-69faca9b28d9 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:46.688829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d85bd6d-c0d6-44ec-9250-69faca9b28d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:46.689207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d85bd6d-c0d6-44ec-9250-69faca9b28d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.690038 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d85bd6d-c0d6-44ec-9250-69faca9b28d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:46.692538 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d85bd6d-c0d6-44ec-9250-69faca9b28d9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:46.731607 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1369/2753] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:46 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:46.734423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5562db3b-6c63-4b98-a19c-b61ec8fb90ad admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:46.747315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5562db3b-6c63-4b98-a19c-b61ec8fb90ad admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:46.759826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3d9f0-3db8-4319-957b-12476e144ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:46.760900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3d9f0-3db8-4319-957b-12476e144ae1 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:46.761384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3d9f0-3db8-4319-957b-12476e144ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:46.761866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3d9f0-3db8-4319-957b-12476e144ae1 admin admin] PATH_INFO: `/v3/projects/c4c4c9cf573c4a4894f1863a68d77df5/users/9d6cd1008c6444e1ac06db27dd911b8f/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:46.793603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa3d9f0-3db8-4319-957b-12476e144ae1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4c4c9cf573c4a4894f1863a68d77df5, user_id=9d6cd1008c6444e1ac06db27dd911b8f, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:46.797714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa3d9f0-3db8-4319-957b-12476e144ae1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:46.827450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1385/2754] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:46 2022] PUT /identity/v3/projects/c4c4c9cf573c4a4894f1863a68d77df5/users/9d6cd1008c6444e1ac06db27dd911b8f/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:46.835779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d85bd6d-c0d6-44ec-9250-69faca9b28d9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:46.877611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d85bd6d-c0d6-44ec-9250-69faca9b28d9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:46.895398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:47.051355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8514111b-d9fe-4386-8880-c6aa8f983789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:47.054508 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8514111b-d9fe-4386-8880-c6aa8f983789 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:47.054921 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8514111b-d9fe-4386-8880-c6aa8f983789 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:47.055297 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8514111b-d9fe-4386-8880-c6aa8f983789 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:47.056644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8514111b-d9fe-4386-8880-c6aa8f983789 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:47.059741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8514111b-d9fe-4386-8880-c6aa8f983789 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:47.080480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1370/2755] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:46 2022] GET /identity/v3/roles => generated 1420 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:47.084564 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3d9f0-3db8-4319-957b-12476e144ae1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:47.101061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3d9f0-3db8-4319-957b-12476e144ae1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:47.107055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:47.108407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:47.108986 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:47.109330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] PATH_INFO: `/v3/projects/c4c4c9cf573c4a4894f1863a68d77df5/users/9d6cd1008c6444e1ac06db27dd911b8f/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:47.112477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4c4c9cf573c4a4894f1863a68d77df5, user_id=9d6cd1008c6444e1ac06db27dd911b8f, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:47.118010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:47.147467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1386/2756] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:47 2022] PUT /identity/v3/projects/c4c4c9cf573c4a4894f1863a68d77df5/users/9d6cd1008c6444e1ac06db27dd911b8f/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:47.160933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f095d19-c866-4e3e-b7ba-36cea4405887 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:47.161567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f095d19-c866-4e3e-b7ba-36cea4405887 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:47.162135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f095d19-c866-4e3e-b7ba-36cea4405887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:47.222772 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-7f095d19-c866-4e3e-b7ba-36cea4405887 None None] MFA Rules not processed for user `9d6cd1008c6444e1ac06db27dd911b8f`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:47.227115 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-7f095d19-c866-4e3e-b7ba-36cea4405887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:47.390920 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1371/2757] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:47 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:47.408297 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:47.412254 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:47.448842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:47.580568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cbce77-4fd2-487a-91f6-dc1705d6bdb7 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:47.589298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8543a2-0dda-46ed-a491-93bde05906f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:47.590361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8543a2-0dda-46ed-a491-93bde05906f7 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:47.590897 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8543a2-0dda-46ed-a491-93bde05906f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:47.592203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8543a2-0dda-46ed-a491-93bde05906f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:47.625010 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8543a2-0dda-46ed-a491-93bde05906f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:47.629930 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8543a2-0dda-46ed-a491-93bde05906f7 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:47.672433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1387/2758] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:47 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:47.831173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f095d19-c866-4e3e-b7ba-36cea4405887 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:47.854937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f095d19-c866-4e3e-b7ba-36cea4405887 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:47.864072 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:47.865376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:47.865831 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:47.867032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:47.958317 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed68d88-5707-457f-8520-6a059778f815 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:47.958926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed68d88-5707-457f-8520-6a059778f815 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:47.959329 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed68d88-5707-457f-8520-6a059778f815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:48.003520 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:48.005360 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:48.018750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-eed68d88-5707-457f-8520-6a059778f815 None None] MFA Rules not processed for user `17a5395280b342de9a39f2a984a2c0ba`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:48.020230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-eed68d88-5707-457f-8520-6a059778f815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:48.042870 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1372/2759] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:47 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:48.207644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1388/2760] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:47 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:48.225233 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:48.227563 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:48.275786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba8a057-e17c-41d2-8cb5-f7b4f4a51b22 service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:48.290299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edfae1a-b95a-48ad-b49a-8b4c8ae1944c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:48.290299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edfae1a-b95a-48ad-b49a-8b4c8ae1944c service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:48.290299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edfae1a-b95a-48ad-b49a-8b4c8ae1944c service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:48.290299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edfae1a-b95a-48ad-b49a-8b4c8ae1944c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:48.320183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edfae1a-b95a-48ad-b49a-8b4c8ae1944c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:48.322082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edfae1a-b95a-48ad-b49a-8b4c8ae1944c service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:48.379289 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1373/2761] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:48 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:48:50.259773 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efc3cf5-93d3-4194-85cf-1a99b0a0141b None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:50.260531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efc3cf5-93d3-4194-85cf-1a99b0a0141b None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:50.260531 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efc3cf5-93d3-4194-85cf-1a99b0a0141b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:50.352162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7efc3cf5-93d3-4194-85cf-1a99b0a0141b None None] MFA Rules not processed for user `17a5395280b342de9a39f2a984a2c0ba`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:50.354135 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7efc3cf5-93d3-4194-85cf-1a99b0a0141b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:50.408104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1389/2762] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:50 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:48:51.060991 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edfae1a-b95a-48ad-b49a-8b4c8ae1944c service cinder] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:51.098290 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:51.250381 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edfae1a-b95a-48ad-b49a-8b4c8ae1944c service cinder] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:51.265388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:51.268767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:51.269126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:51.270257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:51.275777 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:51.298430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:51.301997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:51.326938 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1374/2763] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:48:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:48:54.593841 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d72d172-6725-40dd-8b6f-3145ab7effaf None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:54.598442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d72d172-6725-40dd-8b6f-3145ab7effaf None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:54.598442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d72d172-6725-40dd-8b6f-3145ab7effaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:54.649828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7d72d172-6725-40dd-8b6f-3145ab7effaf None None] MFA Rules not processed for user `9f7d298884b743f49760437628fc65aa`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:54.651628 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7d72d172-6725-40dd-8b6f-3145ab7effaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:54.802794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1390/2764] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:54 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:54.805320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:55.042405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565d6fa1-bcc9-4523-9d4c-4ee5be3e82e6 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:55.063270 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65faa5f4-32b8-4bd8-83bb-9ea948443153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:55.063928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65faa5f4-32b8-4bd8-83bb-9ea948443153 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:55.064209 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65faa5f4-32b8-4bd8-83bb-9ea948443153 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:55.064484 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65faa5f4-32b8-4bd8-83bb-9ea948443153 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:55.065196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65faa5f4-32b8-4bd8-83bb-9ea948443153 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:55.068829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65faa5f4-32b8-4bd8-83bb-9ea948443153 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:55.156958 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1375/2765] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:54 2022] POST /identity/v3/projects => generated 383 bytes in 353 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:55.162839 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d72d172-6725-40dd-8b6f-3145ab7effaf None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:55.169005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:55.180774 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d72d172-6725-40dd-8b6f-3145ab7effaf None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:55.287402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb3b840-0fef-4a27-b693-d63306db882d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:55.290688 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3b840-0fef-4a27-b693-d63306db882d admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:55.290980 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3b840-0fef-4a27-b693-d63306db882d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:55.291512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb3b840-0fef-4a27-b693-d63306db882d admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:55.292310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb3b840-0fef-4a27-b693-d63306db882d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:55.295606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb3b840-0fef-4a27-b693-d63306db882d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:55.330127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1391/2766] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:55 2022] POST /identity/v3/users => generated 298 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:48:55.333470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65faa5f4-32b8-4bd8-83bb-9ea948443153 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:55.347475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65faa5f4-32b8-4bd8-83bb-9ea948443153 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:55.354643 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87eff7-b25d-4d0a-952a-1f2315a751f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:55.355305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87eff7-b25d-4d0a-952a-1f2315a751f0 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:55.355631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87eff7-b25d-4d0a-952a-1f2315a751f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:55.355914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87eff7-b25d-4d0a-952a-1f2315a751f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:55.357575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e87eff7-b25d-4d0a-952a-1f2315a751f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:55.359589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e87eff7-b25d-4d0a-952a-1f2315a751f0 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:55.382348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1376/2767] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:55 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:55.388421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb3b840-0fef-4a27-b693-d63306db882d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:55.415002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb3b840-0fef-4a27-b693-d63306db882d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:55.422537 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c617d594-a3cf-4efe-ac56-a9413adec33a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:55.423492 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c617d594-a3cf-4efe-ac56-a9413adec33a admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:55.423885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c617d594-a3cf-4efe-ac56-a9413adec33a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:55.424263 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c617d594-a3cf-4efe-ac56-a9413adec33a admin admin] PATH_INFO: `/v3/projects/3738ca011675489ba88c4265e0f1488b/users/7f7914881e2646229ee619ed5c96666e/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:55.471868 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c617d594-a3cf-4efe-ac56-a9413adec33a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3738ca011675489ba88c4265e0f1488b, user_id=7f7914881e2646229ee619ed5c96666e, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:55.476341 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c617d594-a3cf-4efe-ac56-a9413adec33a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:55.510460 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1392/2768] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:55 2022] PUT /identity/v3/projects/3738ca011675489ba88c4265e0f1488b/users/7f7914881e2646229ee619ed5c96666e/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:55.511851 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87eff7-b25d-4d0a-952a-1f2315a751f0 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:55.529916 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87eff7-b25d-4d0a-952a-1f2315a751f0 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:55.643807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda22557-8111-472b-be9c-c6e655df15da None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:55.644379 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda22557-8111-472b-be9c-c6e655df15da None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:55.644480 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda22557-8111-472b-be9c-c6e655df15da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:55.653442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:55.691643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-dda22557-8111-472b-be9c-c6e655df15da None None] MFA Rules not processed for user `1addff9d7c7348de827c4eba89d41660`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:48:55.692887 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-dda22557-8111-472b-be9c-c6e655df15da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:48:55.693256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff94e056-1c5a-49a8-914a-e8d122ded74b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:55.694174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94e056-1c5a-49a8-914a-e8d122ded74b admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:55.694622 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94e056-1c5a-49a8-914a-e8d122ded74b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:55.695093 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94e056-1c5a-49a8-914a-e8d122ded74b admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:55.695947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff94e056-1c5a-49a8-914a-e8d122ded74b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:55.698309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff94e056-1c5a-49a8-914a-e8d122ded74b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:55.713145 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1377/2769] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:55 2022] GET /identity/v3/roles => generated 1420 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:55.718452 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff94e056-1c5a-49a8-914a-e8d122ded74b admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:55.729876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff94e056-1c5a-49a8-914a-e8d122ded74b admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:55.736258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3da1de5-6a92-4791-8c11-333acb5d3f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:55.737443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da1de5-6a92-4791-8c11-333acb5d3f4b admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:55.737619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da1de5-6a92-4791-8c11-333acb5d3f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:55.737908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da1de5-6a92-4791-8c11-333acb5d3f4b admin admin] PATH_INFO: `/v3/projects/3738ca011675489ba88c4265e0f1488b/users/7f7914881e2646229ee619ed5c96666e/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:55.740109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3da1de5-6a92-4791-8c11-333acb5d3f4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3738ca011675489ba88c4265e0f1488b, user_id=7f7914881e2646229ee619ed5c96666e, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:55.745611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3da1de5-6a92-4791-8c11-333acb5d3f4b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:55.761163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1378/2770] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:55 2022] PUT /identity/v3/projects/3738ca011675489ba88c4265e0f1488b/users/7f7914881e2646229ee619ed5c96666e/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:55.830780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1393/2771] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:48:55 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:48:55.834014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3da1de5-6a92-4791-8c11-333acb5d3f4b admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:55.997541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3da1de5-6a92-4791-8c11-333acb5d3f4b admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:56.013770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173e9ff-bd41-493a-ab08-91ce03554b62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:56.016332 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173e9ff-bd41-493a-ab08-91ce03554b62 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:56.018135 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173e9ff-bd41-493a-ab08-91ce03554b62 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:56.018966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173e9ff-bd41-493a-ab08-91ce03554b62 admin admin] PATH_INFO: `/v3/projects` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:56.020656 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e173e9ff-bd41-493a-ab08-91ce03554b62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:56.025694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e173e9ff-bd41-493a-ab08-91ce03554b62 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:56.095100 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1379/2772] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:48:55 2022] POST /identity/v3/projects => generated 377 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:56.099900 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda22557-8111-472b-be9c-c6e655df15da None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:56.121267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda22557-8111-472b-be9c-c6e655df15da None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:56.240113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:48:56.259169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16faa50-0b6a-4024-8fe6-40f65a0542fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:56.274641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16faa50-0b6a-4024-8fe6-40f65a0542fb admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:56.276019 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16faa50-0b6a-4024-8fe6-40f65a0542fb admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:56.277123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16faa50-0b6a-4024-8fe6-40f65a0542fb admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:56.278000 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16faa50-0b6a-4024-8fe6-40f65a0542fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:56.282882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16faa50-0b6a-4024-8fe6-40f65a0542fb admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:56.316218 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1394/2773] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:48:56 2022] POST /identity/v3/users => generated 295 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:48:56.321483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173e9ff-bd41-493a-ab08-91ce03554b62 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:56.338278 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173e9ff-bd41-493a-ab08-91ce03554b62 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:56.345879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3bd7-5c1e-403d-a9bb-7551e4cfef47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:56.346409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3bd7-5c1e-403d-a9bb-7551e4cfef47 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:56.349677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3bd7-5c1e-403d-a9bb-7551e4cfef47 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:56.350006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb3bd7-5c1e-403d-a9bb-7551e4cfef47 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:56.350659 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb3bd7-5c1e-403d-a9bb-7551e4cfef47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:56.352355 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbb3bd7-5c1e-403d-a9bb-7551e4cfef47 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:56.380684 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1380/2774] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:56 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:56.382656 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16faa50-0b6a-4024-8fe6-40f65a0542fb admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:56.397368 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16faa50-0b6a-4024-8fe6-40f65a0542fb admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:56.405355 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1b1942-c907-404a-b718-afed3547a6bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:56.406693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b1942-c907-404a-b718-afed3547a6bf admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:56.407026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b1942-c907-404a-b718-afed3547a6bf admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:56.407308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b1942-c907-404a-b718-afed3547a6bf admin admin] PATH_INFO: `/v3/projects/7e728c6888fe4d159900e8212c6c2e81/users/576a13523a5d4cbba20fbd95c183a173/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:56.430801 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1b1942-c907-404a-b718-afed3547a6bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e728c6888fe4d159900e8212c6c2e81, user_id=576a13523a5d4cbba20fbd95c183a173, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:56.437797 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1b1942-c907-404a-b718-afed3547a6bf admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:56.483434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1395/2775] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:56 2022] PUT /identity/v3/projects/7e728c6888fe4d159900e8212c6c2e81/users/576a13523a5d4cbba20fbd95c183a173/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:48:56.491030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3bd7-5c1e-403d-a9bb-7551e4cfef47 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:56.508357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb3bd7-5c1e-403d-a9bb-7551e4cfef47 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:56.702821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e00e6b4-6606-465d-a737-588ae5f16881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:56.703826 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00e6b4-6606-465d-a737-588ae5f16881 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:56.704212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00e6b4-6606-465d-a737-588ae5f16881 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:56.704592 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00e6b4-6606-465d-a737-588ae5f16881 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:56.705375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e00e6b4-6606-465d-a737-588ae5f16881 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:56.707639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e00e6b4-6606-465d-a737-588ae5f16881 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:56.741274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1381/2776] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:48:56 2022] GET /identity/v3/roles => generated 1420 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:48:56.745933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1b1942-c907-404a-b718-afed3547a6bf admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:48:56.770690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1b1942-c907-404a-b718-afed3547a6bf admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:48:56.793965 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3743dd-d1b6-4614-8054-007c7f25b616 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:48:56.795529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3743dd-d1b6-4614-8054-007c7f25b616 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:48:56.796207 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3743dd-d1b6-4614-8054-007c7f25b616 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:48:56.796319 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3743dd-d1b6-4614-8054-007c7f25b616 admin admin] PATH_INFO: `/v3/projects/7e728c6888fe4d159900e8212c6c2e81/users/576a13523a5d4cbba20fbd95c183a173/roles/38171c89968943bd92d9e9758db65615` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:48:56.800727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3743dd-d1b6-4614-8054-007c7f25b616 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e728c6888fe4d159900e8212c6c2e81, user_id=576a13523a5d4cbba20fbd95c183a173, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:48:56.813515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3743dd-d1b6-4614-8054-007c7f25b616 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:48:56.870201 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1396/2777] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:48:56 2022] PUT /identity/v3/projects/7e728c6888fe4d159900e8212c6c2e81/users/576a13523a5d4cbba20fbd95c183a173/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:00.568446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e00e6b4-6606-465d-a737-588ae5f16881 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:00.582163 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e00e6b4-6606-465d-a737-588ae5f16881 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:00.674111 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:00.729857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:00.730503 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:00.730996 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:00.731482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] PATH_INFO: `/v3/users/953f4b84484541e4a33365cd01788be0` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:00.734513 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=953f4b84484541e4a33365cd01788be0)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:00.741500 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:00.915284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '953f4b84484541e4a33365cd01788be0'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:00.916529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '953f4b84484541e4a33365cd01788be0'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:00.938212 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '953f4b84484541e4a33365cd01788be0'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:00.953709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '953f4b84484541e4a33365cd01788be0'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:00.977471 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '953f4b84484541e4a33365cd01788be0'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:00.987022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1382/2778] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:49:00 2022] DELETE /identity/v3/users/953f4b84484541e4a33365cd01788be0 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:00.993020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9817af-713f-4ec1-8ad5-e386ed53ac7c None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:00.993544 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9817af-713f-4ec1-8ad5-e386ed53ac7c None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:00.994155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9817af-713f-4ec1-8ad5-e386ed53ac7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:01.078123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-8b9817af-713f-4ec1-8ad5-e386ed53ac7c None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:01.092211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8b9817af-713f-4ec1-8ad5-e386ed53ac7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:01.281029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1397/2779] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:01.296002 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:01.302430 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:01.403102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8062a6-0836-446f-9394-18f96415f0ca admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:01.410997 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdf5f01-d1dc-4e0f-9b4b-30820152033f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:01.411635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdf5f01-d1dc-4e0f-9b4b-30820152033f service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:01.411911 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdf5f01-d1dc-4e0f-9b4b-30820152033f service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:01.412181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdf5f01-d1dc-4e0f-9b4b-30820152033f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:01.444457 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdf5f01-d1dc-4e0f-9b4b-30820152033f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:01.449469 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdf5f01-d1dc-4e0f-9b4b-30820152033f service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:01.483590 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1383/2780] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:02.147089 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9817af-713f-4ec1-8ad5-e386ed53ac7c None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:02.156495 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-8b9817af-713f-4ec1-8ad5-e386ed53ac7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:02.188872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9817af-713f-4ec1-8ad5-e386ed53ac7c None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:02.209394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:02.211551 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:02.211936 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:02.213223 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] PATH_INFO: `/v3/projects/fb29932410eb4588868635137f669d23` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:02.215156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb29932410eb4588868635137f669d23)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:02.218667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:02.311584 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb29932410eb4588868635137f669d23', 'request_id': 'req-9dabaa01-84b5-450a-8c40-0accfb50ebbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.325652 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb29932410eb4588868635137f669d23', 'request_id': 'req-9dabaa01-84b5-450a-8c40-0accfb50ebbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.334414 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb29932410eb4588868635137f669d23', 'request_id': 'req-9dabaa01-84b5-450a-8c40-0accfb50ebbe'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.372285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] The token cache is being invalidate because project fb29932410eb4588868635137f669d23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:49:02.372285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.373600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.377172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1398/2781] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:49:02 2022] DELETE /identity/v3/projects/fb29932410eb4588868635137f669d23 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:02.384528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdf5f01-d1dc-4e0f-9b4b-30820152033f service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:02.423468 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdf5f01-d1dc-4e0f-9b4b-30820152033f service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:02.561128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:02.590977 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:02.592189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:02.592480 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:02.592809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] PATH_INFO: `/v3/users/9d6cd1008c6444e1ac06db27dd911b8f` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:02.594371 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d6cd1008c6444e1ac06db27dd911b8f)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:02.597944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:02.826344 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d6cd1008c6444e1ac06db27dd911b8f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.827147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d6cd1008c6444e1ac06db27dd911b8f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.834928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d6cd1008c6444e1ac06db27dd911b8f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.851929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d6cd1008c6444e1ac06db27dd911b8f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.876309 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-2bae30e7-98bf-4208-aaef-b29590b8206d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d6cd1008c6444e1ac06db27dd911b8f'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:02.886928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1384/2782] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:49:02 2022] DELETE /identity/v3/users/9d6cd1008c6444e1ac06db27dd911b8f => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:03.446304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:03.447809 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:03.470482 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dabaa01-84b5-450a-8c40-0accfb50ebbe admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:03.548136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=87,keystone:INSERT=6,keystone:DELETE=7 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:03.641693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:03.643119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:03.643664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:03.644296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] PATH_INFO: `/v3/projects/c4c4c9cf573c4a4894f1863a68d77df5` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:03.646109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4c4c9cf573c4a4894f1863a68d77df5)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:03.650646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:03.742301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4c4c9cf573c4a4894f1863a68d77df5', 'request_id': 'req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:03.752543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4c4c9cf573c4a4894f1863a68d77df5', 'request_id': 'req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:03.767772 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4c4c9cf573c4a4894f1863a68d77df5', 'request_id': 'req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:03.797383 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] The token cache is being invalidate because project c4c4c9cf573c4a4894f1863a68d77df5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:49:03.797954 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:03.798472 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-223f26d3-e15b-4343-bba4-d4dabe1b1aa2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:03.806794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1399/2783] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:49:03 2022] DELETE /identity/v3/projects/c4c4c9cf573c4a4894f1863a68d77df5 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:03.808733 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c565655-520f-45e7-83ae-9f712d019b93 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:03.809438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c565655-520f-45e7-83ae-9f712d019b93 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:03.809438 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c565655-520f-45e7-83ae-9f712d019b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:03.895271 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1c565655-520f-45e7-83ae-9f712d019b93 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:03.896722 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1c565655-520f-45e7-83ae-9f712d019b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:03.997966 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1385/2784] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:04.014313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58c235a-a35f-4716-a7a2-30d8df27a9dc None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:04.014605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58c235a-a35f-4716-a7a2-30d8df27a9dc None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:04.015162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58c235a-a35f-4716-a7a2-30d8df27a9dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:04.048115 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e58c235a-a35f-4716-a7a2-30d8df27a9dc None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:04.049595 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e58c235a-a35f-4716-a7a2-30d8df27a9dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:04.062968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1400/2785] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:49:04.064215 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c565655-520f-45e7-83ae-9f712d019b93 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:04.083694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c565655-520f-45e7-83ae-9f712d019b93 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:04.088847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:04.089567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:04.090059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:04.090402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:04.091574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:49:04.091775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:04.094481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:04.109779 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1386/2786] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:49:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:49:04.116459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58c235a-a35f-4716-a7a2-30d8df27a9dc None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:04.125405 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58c235a-a35f-4716-a7a2-30d8df27a9dc None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:04.132516 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3ff92b-824c-4564-9514-1da7ce6d4947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:04.133253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3ff92b-824c-4564-9514-1da7ce6d4947 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:04.133596 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3ff92b-824c-4564-9514-1da7ce6d4947 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:04.133941 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3ff92b-824c-4564-9514-1da7ce6d4947 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:04.134692 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3ff92b-824c-4564-9514-1da7ce6d4947 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:04.136766 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3ff92b-824c-4564-9514-1da7ce6d4947 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:04.171912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1401/2787] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:49:04 2022] POST /identity/v3/projects => generated 407 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:04.177877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:04.192043 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062044a-e95b-4bbd-b151-7357770cbf77 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:04.240810 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:04.280579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0235df49-4fdd-49f1-8742-e678d9aa78eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:04.281595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235df49-4fdd-49f1-8742-e678d9aa78eb admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:04.282362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235df49-4fdd-49f1-8742-e678d9aa78eb admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:04.282993 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235df49-4fdd-49f1-8742-e678d9aa78eb admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:04.284028 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0235df49-4fdd-49f1-8742-e678d9aa78eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:04.286624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0235df49-4fdd-49f1-8742-e678d9aa78eb admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:04.322113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1387/2788] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:49:04 2022] POST /identity/v3/users => generated 304 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:04.326156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3ff92b-824c-4564-9514-1da7ce6d4947 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:04.338162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3ff92b-824c-4564-9514-1da7ce6d4947 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:04.344736 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850c8ef1-72a5-4ad9-8b8a-540fe81f398f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:04.345573 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850c8ef1-72a5-4ad9-8b8a-540fe81f398f admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:04.346939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850c8ef1-72a5-4ad9-8b8a-540fe81f398f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:04.346939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850c8ef1-72a5-4ad9-8b8a-540fe81f398f admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:04.347186 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850c8ef1-72a5-4ad9-8b8a-540fe81f398f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:04.349481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850c8ef1-72a5-4ad9-8b8a-540fe81f398f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:04.361737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1402/2789] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:04 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:49:04.367109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0235df49-4fdd-49f1-8742-e678d9aa78eb admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:04.389725 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0235df49-4fdd-49f1-8742-e678d9aa78eb admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:04.394652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d807a7-fe43-4665-810d-ac7fc9a6d23b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:04.395299 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d807a7-fe43-4665-810d-ac7fc9a6d23b admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:04.395581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d807a7-fe43-4665-810d-ac7fc9a6d23b admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:04.395891 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d807a7-fe43-4665-810d-ac7fc9a6d23b admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:04.396521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d807a7-fe43-4665-810d-ac7fc9a6d23b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:04.398302 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d807a7-fe43-4665-810d-ac7fc9a6d23b admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:04.421509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1388/2790] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:04 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:04.426169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850c8ef1-72a5-4ad9-8b8a-540fe81f398f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:04.456687 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850c8ef1-72a5-4ad9-8b8a-540fe81f398f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:04.477501 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee906b5-8155-470b-8c6c-ab03a50f1b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:04.478859 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee906b5-8155-470b-8c6c-ab03a50f1b3a admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:04.479157 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee906b5-8155-470b-8c6c-ab03a50f1b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:04.479443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee906b5-8155-470b-8c6c-ab03a50f1b3a admin admin] PATH_INFO: `/v3/projects/aa029c94a8824c6ca80fa741f3f36838/users/736d560530a34aa1a063d4947edaa011/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:04.519215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee906b5-8155-470b-8c6c-ab03a50f1b3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa029c94a8824c6ca80fa741f3f36838, user_id=736d560530a34aa1a063d4947edaa011, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:04.522685 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee906b5-8155-470b-8c6c-ab03a50f1b3a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:04.569062 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66061a81-05fd-4c2a-8427-83c4a8ebadd9 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:04.569537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66061a81-05fd-4c2a-8427-83c4a8ebadd9 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:04.570804 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66061a81-05fd-4c2a-8427-83c4a8ebadd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:04.573740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1403/2791] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:49:04 2022] PUT /identity/v3/projects/aa029c94a8824c6ca80fa741f3f36838/users/736d560530a34aa1a063d4947edaa011/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:04.652486 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-66061a81-05fd-4c2a-8427-83c4a8ebadd9 None None] MFA Rules not processed for user `736d560530a34aa1a063d4947edaa011`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:04.653821 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-66061a81-05fd-4c2a-8427-83c4a8ebadd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:04.866558 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1389/2792] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:04 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:04.872920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee906b5-8155-470b-8c6c-ab03a50f1b3a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:04.968600 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:04.984933 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee906b5-8155-470b-8c6c-ab03a50f1b3a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:04.995901 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e219257-5976-4a15-af74-39703773e7a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:04.999849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e219257-5976-4a15-af74-39703773e7a8 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:05.001145 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e219257-5976-4a15-af74-39703773e7a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:05.002328 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e219257-5976-4a15-af74-39703773e7a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:05.003684 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e219257-5976-4a15-af74-39703773e7a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:05.006770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e219257-5976-4a15-af74-39703773e7a8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:05.081960 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66061a81-05fd-4c2a-8427-83c4a8ebadd9 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:05.095346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1404/2793] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:49:04 2022] POST /identity/v3/projects => generated 409 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:05.123807 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66061a81-05fd-4c2a-8427-83c4a8ebadd9 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:05.315824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9976db01-a694-4b40-b0e2-d869bb2d4a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:05.316611 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9976db01-a694-4b40-b0e2-d869bb2d4a0a admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:05.317322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9976db01-a694-4b40-b0e2-d869bb2d4a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:05.317913 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9976db01-a694-4b40-b0e2-d869bb2d4a0a admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:05.318767 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9976db01-a694-4b40-b0e2-d869bb2d4a0a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:05.321667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9976db01-a694-4b40-b0e2-d869bb2d4a0a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:05.366040 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e219257-5976-4a15-af74-39703773e7a8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:05.366338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1390/2794] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:49:05 2022] POST /identity/v3/users => generated 311 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:05.387953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e219257-5976-4a15-af74-39703773e7a8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:05.409837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ef87f8-f6ea-43d8-ad3f-633f29631760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:05.411393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ef87f8-f6ea-43d8-ad3f-633f29631760 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:05.412196 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ef87f8-f6ea-43d8-ad3f-633f29631760 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:05.412978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ef87f8-f6ea-43d8-ad3f-633f29631760 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:05.414466 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ef87f8-f6ea-43d8-ad3f-633f29631760 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:05.420417 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ef87f8-f6ea-43d8-ad3f-633f29631760 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:05.445374 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9976db01-a694-4b40-b0e2-d869bb2d4a0a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:05.446103 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1405/2795] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:05 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:05.466529 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9976db01-a694-4b40-b0e2-d869bb2d4a0a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:05.483263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cdf73c-f059-44b0-a524-3edd19787b32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:05.484475 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cdf73c-f059-44b0-a524-3edd19787b32 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:05.484872 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cdf73c-f059-44b0-a524-3edd19787b32 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:05.485258 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cdf73c-f059-44b0-a524-3edd19787b32 admin admin] PATH_INFO: `/v3/projects/94489c90fc544dff87ec0ea624172c18/users/68be20a3c8204afb8c9fa2adcaf58650/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:05.539770 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cdf73c-f059-44b0-a524-3edd19787b32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94489c90fc544dff87ec0ea624172c18, user_id=68be20a3c8204afb8c9fa2adcaf58650, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:05.545268 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cdf73c-f059-44b0-a524-3edd19787b32 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:05.593284 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1391/2796] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:49:05 2022] PUT /identity/v3/projects/94489c90fc544dff87ec0ea624172c18/users/68be20a3c8204afb8c9fa2adcaf58650/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:05.597244 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ef87f8-f6ea-43d8-ad3f-633f29631760 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:05.610623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ef87f8-f6ea-43d8-ad3f-633f29631760 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:05.843720 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b3b505-8134-4835-b0e0-54ef5c0c15dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:05.844591 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3b505-8134-4835-b0e0-54ef5c0c15dc admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:05.844993 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3b505-8134-4835-b0e0-54ef5c0c15dc admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:05.845425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3b505-8134-4835-b0e0-54ef5c0c15dc admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:05.854621 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b3b505-8134-4835-b0e0-54ef5c0c15dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:05.861100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b3b505-8134-4835-b0e0-54ef5c0c15dc admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:05.911333 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1406/2797] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:05 2022] GET /identity/v3/roles => generated 1420 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:05.916117 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cdf73c-f059-44b0-a524-3edd19787b32 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:05.940455 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cdf73c-f059-44b0-a524-3edd19787b32 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:05.956463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:05.958620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:05.958936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:05.960433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] PATH_INFO: `/v3/projects/94489c90fc544dff87ec0ea624172c18/users/68be20a3c8204afb8c9fa2adcaf58650/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:05.966691 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94489c90fc544dff87ec0ea624172c18, user_id=68be20a3c8204afb8c9fa2adcaf58650, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:05.972463 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:05.991694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:06.014306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1392/2798] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:49:05 2022] PUT /identity/v3/projects/94489c90fc544dff87ec0ea624172c18/users/68be20a3c8204afb8c9fa2adcaf58650/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:49:06.026170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7419408c-af51-48a2-a9c5-1d3b00d68c7e None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:06.026525 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7419408c-af51-48a2-a9c5-1d3b00d68c7e None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:06.026920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7419408c-af51-48a2-a9c5-1d3b00d68c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:06.127945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-7419408c-af51-48a2-a9c5-1d3b00d68c7e None None] MFA Rules not processed for user `68be20a3c8204afb8c9fa2adcaf58650`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:06.132219 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-7419408c-af51-48a2-a9c5-1d3b00d68c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:06.394641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1407/2799] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:06 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 366 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:06.409667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:06.412499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:06.628397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5d7820-6730-4545-b717-b1be7b6094a0 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:06.646701 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1f8ec6-a1eb-4302-822b-c0fb1a2d6ea9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:06.647369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f8ec6-a1eb-4302-822b-c0fb1a2d6ea9 service cinder] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:06.647652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f8ec6-a1eb-4302-822b-c0fb1a2d6ea9 service cinder] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:06.647926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f8ec6-a1eb-4302-822b-c0fb1a2d6ea9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:06.668641 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1f8ec6-a1eb-4302-822b-c0fb1a2d6ea9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:06.671047 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1f8ec6-a1eb-4302-822b-c0fb1a2d6ea9 service cinder] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:06.714091 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1393/2800] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:06 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:49:07.095792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7419408c-af51-48a2-a9c5-1d3b00d68c7e None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:07.113237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7419408c-af51-48a2-a9c5-1d3b00d68c7e None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:07.119054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:07.123846 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:07.124365 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:07.124793 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:07.301326 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:07.388278 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:07.408529 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:07.438098 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1408/2801] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:07 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:49:08.952351 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510b70b-9206-4e11-ae55-0c044819083f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:08.953652 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510b70b-9206-4e11-ae55-0c044819083f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:08.954952 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510b70b-9206-4e11-ae55-0c044819083f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:09.011504 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-9510b70b-9206-4e11-ae55-0c044819083f None None] MFA Rules not processed for user `576a13523a5d4cbba20fbd95c183a173`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:09.012788 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-9510b70b-9206-4e11-ae55-0c044819083f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:09.125347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1394/2802] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:08 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:09.140842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:09.143318 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:09.231126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376f5263-ad1f-464d-96a3-d3a5bd913a43 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:09.231527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376f5263-ad1f-464d-96a3-d3a5bd913a43 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:09.231880 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376f5263-ad1f-464d-96a3-d3a5bd913a43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:09.283399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-376f5263-ad1f-464d-96a3-d3a5bd913a43 None None] MFA Rules not processed for user `7f7914881e2646229ee619ed5c96666e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:09.285605 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-376f5263-ad1f-464d-96a3-d3a5bd913a43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:09.318424 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c87ef5e-cd18-4e5f-81ce-b97f0597c903 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:09.327078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324618be-53bc-40fb-8301-723ef5ca3236 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:09.328162 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324618be-53bc-40fb-8301-723ef5ca3236 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:09.344421 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324618be-53bc-40fb-8301-723ef5ca3236 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:09.344768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324618be-53bc-40fb-8301-723ef5ca3236 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:09.349178 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-324618be-53bc-40fb-8301-723ef5ca3236 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:09.444534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1395/2803] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:09 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:49:09.469140 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376f5263-ad1f-464d-96a3-d3a5bd913a43 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:09.471848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-376f5263-ad1f-464d-96a3-d3a5bd913a43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:09.494051 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324618be-53bc-40fb-8301-723ef5ca3236 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:09.496173 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324618be-53bc-40fb-8301-723ef5ca3236 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:09.500006 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:09.519705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1409/2804] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:09 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:09.649741 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376f5263-ad1f-464d-96a3-d3a5bd913a43 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:09.652067 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:09.661930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:09.662236 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:09.662539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:09.676442 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:09.939032 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:09.942010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:09.982851 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1396/2805] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:09 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 514 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:10.197086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8deae21-dbf0-42c7-adf4-83637ef01205 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:10.197707 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8deae21-dbf0-42c7-adf4-83637ef01205 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:10.197858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8deae21-dbf0-42c7-adf4-83637ef01205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:10.285637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-e8deae21-dbf0-42c7-adf4-83637ef01205 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:10.287231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e8deae21-dbf0-42c7-adf4-83637ef01205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:10.458387 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=100 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:10.485951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1410/2806] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:49:10 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:10.583511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:10.585066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:10.761040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2603cdc-2639-4ae9-92f4-e4e433c1714b service glance] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:10.774983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72e57ee-5b49-432e-87b0-2ef45a80b513 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:10.776044 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72e57ee-5b49-432e-87b0-2ef45a80b513 service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:10.776555 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72e57ee-5b49-432e-87b0-2ef45a80b513 service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:10.777173 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72e57ee-5b49-432e-87b0-2ef45a80b513 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:10.819051 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72e57ee-5b49-432e-87b0-2ef45a80b513 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:10.823206 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72e57ee-5b49-432e-87b0-2ef45a80b513 service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:10.887786 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1397/2807] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:49:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:49:16.997915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8deae21-dbf0-42c7-adf4-83637ef01205 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:17.181374 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8deae21-dbf0-42c7-adf4-83637ef01205 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:17.189995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:17.191189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:17.192005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:17.192603 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] PATH_INFO: `/v3/users/736d560530a34aa1a063d4947edaa011` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:17.197288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] RBAC: Authorizing `identity:delete_user(user_id=736d560530a34aa1a063d4947edaa011)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:17.199740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:17.327420 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '736d560530a34aa1a063d4947edaa011'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:17.345249 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '736d560530a34aa1a063d4947edaa011'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:17.346571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '736d560530a34aa1a063d4947edaa011'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:17.347474 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '736d560530a34aa1a063d4947edaa011'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:17.372057 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '736d560530a34aa1a063d4947edaa011'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:17.432979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1411/2808] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:49:16 2022] DELETE /identity/v3/users/736d560530a34aa1a063d4947edaa011 => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:17.438158 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319856a7-626a-41c4-863f-3f16252d0fdc None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:17.438775 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319856a7-626a-41c4-863f-3f16252d0fdc None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:17.439097 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319856a7-626a-41c4-863f-3f16252d0fdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:17.466537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-319856a7-626a-41c4-863f-3f16252d0fdc None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:17.469679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-319856a7-626a-41c4-863f-3f16252d0fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:17.573296 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=100 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:17.588060 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1398/2809] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:49:17.604982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:17.604982 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:17.790524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bb25be-a99e-4829-828a-f4bac26afc5c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:17.796277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869c46ac-c5af-4635-8bb3-c4ae04576585 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:17.797138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869c46ac-c5af-4635-8bb3-c4ae04576585 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:17.797592 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869c46ac-c5af-4635-8bb3-c4ae04576585 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:17.798029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869c46ac-c5af-4635-8bb3-c4ae04576585 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:17.806369 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:17.812795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869c46ac-c5af-4635-8bb3-c4ae04576585 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:17.815650 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869c46ac-c5af-4635-8bb3-c4ae04576585 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:17.828337 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1412/2810] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:18.376594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319856a7-626a-41c4-863f-3f16252d0fdc None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:18.380680 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-319856a7-626a-41c4-863f-3f16252d0fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:18.451639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319856a7-626a-41c4-863f-3f16252d0fdc None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:18.474426 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:18.475698 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:18.476164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:18.476825 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] PATH_INFO: `/v3/projects/aa029c94a8824c6ca80fa741f3f36838` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:18.494837 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa029c94a8824c6ca80fa741f3f36838)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:18.505507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:18.769556 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa029c94a8824c6ca80fa741f3f36838', 'request_id': 'req-af0c2a91-f435-422b-8b2e-813196faca9a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:18.782514 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa029c94a8824c6ca80fa741f3f36838', 'request_id': 'req-af0c2a91-f435-422b-8b2e-813196faca9a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:18.800429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa029c94a8824c6ca80fa741f3f36838', 'request_id': 'req-af0c2a91-f435-422b-8b2e-813196faca9a'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:18.841262 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] The token cache is being invalidate because project aa029c94a8824c6ca80fa741f3f36838 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:49:18.842010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:18.844305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:18.852391 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1399/2811] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:49:18 2022] DELETE /identity/v3/projects/aa029c94a8824c6ca80fa741f3f36838 => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:18.857892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869c46ac-c5af-4635-8bb3-c4ae04576585 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:18.902253 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869c46ac-c5af-4635-8bb3-c4ae04576585 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:19.005606 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:19.006239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:19.006522 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:19.006820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] PATH_INFO: `/v3/users/68be20a3c8204afb8c9fa2adcaf58650` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:19.007972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] RBAC: Authorizing `identity:delete_user(user_id=68be20a3c8204afb8c9fa2adcaf58650)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:19.009915 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:19.115733 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '68be20a3c8204afb8c9fa2adcaf58650'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:19.131689 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '68be20a3c8204afb8c9fa2adcaf58650'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:19.132975 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '68be20a3c8204afb8c9fa2adcaf58650'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:19.134184 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '68be20a3c8204afb8c9fa2adcaf58650'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:19.169571 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '68be20a3c8204afb8c9fa2adcaf58650'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:19.236140 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1413/2812] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:49:18 2022] DELETE /identity/v3/users/68be20a3c8204afb8c9fa2adcaf58650 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:19.878372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:19.883885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:20.150411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0c2a91-f435-422b-8b2e-813196faca9a admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:20.186402 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:20.193097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:20.307126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:20.330321 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ceefa0-6f1f-40fd-aa00-d0f8baa72e98 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:20.331761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ceefa0-6f1f-40fd-aa00-d0f8baa72e98 service placement] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:20.332144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ceefa0-6f1f-40fd-aa00-d0f8baa72e98 service placement] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:20.332524 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ceefa0-6f1f-40fd-aa00-d0f8baa72e98 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:20.338305 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-68ceefa0-6f1f-40fd-aa00-d0f8baa72e98 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:20.355530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ceefa0-6f1f-40fd-aa00-d0f8baa72e98 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:20.358216 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ceefa0-6f1f-40fd-aa00-d0f8baa72e98 service placement] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:20.398314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1400/2813] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 520 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:20.462703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e7adeb-a3ac-43d8-9e63-c63a9521244d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:20.475300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:20.476407 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:20.476705 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:20.477170 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] PATH_INFO: `/v3/projects/94489c90fc544dff87ec0ea624172c18` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:20.479172 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94489c90fc544dff87ec0ea624172c18)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:20.482711 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:20.539037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:20.664156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94489c90fc544dff87ec0ea624172c18', 'request_id': 'req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:20.680307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94489c90fc544dff87ec0ea624172c18', 'request_id': 'req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:20.695048 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94489c90fc544dff87ec0ea624172c18', 'request_id': 'req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:20.733285 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] The token cache is being invalidate because project 94489c90fc544dff87ec0ea624172c18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:49:20.735845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:20.737009 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-f0185288-fc27-47f8-be9f-2ae52fb6bfe1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:20.740548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1414/2814] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:49:20 2022] DELETE /identity/v3/projects/94489c90fc544dff87ec0ea624172c18 => generated 0 bytes in 555 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:20.754251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74551649-0156-435d-8fa0-c29f91b9770f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:20.755116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74551649-0156-435d-8fa0-c29f91b9770f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:20.755549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74551649-0156-435d-8fa0-c29f91b9770f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:20.795618 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-74551649-0156-435d-8fa0-c29f91b9770f None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:20.831470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-74551649-0156-435d-8fa0-c29f91b9770f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:20.980207 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1401/2815] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:20.991475 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a8482-6359-402d-aa87-009ddf17b281 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:20.991778 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a8482-6359-402d-aa87-009ddf17b281 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:20.992037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a8482-6359-402d-aa87-009ddf17b281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.022033 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-713a8482-6359-402d-aa87-009ddf17b281 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:21.024926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-713a8482-6359-402d-aa87-009ddf17b281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:21.040169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1415/2816] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:21.044152 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74551649-0156-435d-8fa0-c29f91b9770f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:21.059033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74551649-0156-435d-8fa0-c29f91b9770f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:21.066836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056d748d-0ce5-49c5-814a-1a79f8e04842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:21.066836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056d748d-0ce5-49c5-814a-1a79f8e04842 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:21.067177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056d748d-0ce5-49c5-814a-1a79f8e04842 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:21.067450 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056d748d-0ce5-49c5-814a-1a79f8e04842 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.068369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056d748d-0ce5-49c5-814a-1a79f8e04842 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:49:21.068926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056d748d-0ce5-49c5-814a-1a79f8e04842 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:21.083545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056d748d-0ce5-49c5-814a-1a79f8e04842 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:21.118884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1402/2817] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:49:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:49:21.126545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a8482-6359-402d-aa87-009ddf17b281 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:21.149862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a8482-6359-402d-aa87-009ddf17b281 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:21.156568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf296099-78c4-4ad6-9daa-eed39a6e90a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:21.157636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf296099-78c4-4ad6-9daa-eed39a6e90a8 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:21.158258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf296099-78c4-4ad6-9daa-eed39a6e90a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:21.158574 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf296099-78c4-4ad6-9daa-eed39a6e90a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.159693 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf296099-78c4-4ad6-9daa-eed39a6e90a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:21.172518 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf296099-78c4-4ad6-9daa-eed39a6e90a8 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:21.206399 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79ebc8-f65d-4371-955d-464b2431418d None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:21.206956 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79ebc8-f65d-4371-955d-464b2431418d None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:21.207476 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79ebc8-f65d-4371-955d-464b2431418d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.233058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-2a79ebc8-f65d-4371-955d-464b2431418d None None] MFA Rules not processed for user `17a5395280b342de9a39f2a984a2c0ba`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:21.234627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-2a79ebc8-f65d-4371-955d-464b2431418d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:21.260169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1416/2818] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:49:21 2022] POST /identity/v3/projects => generated 377 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:21.265597 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf296099-78c4-4ad6-9daa-eed39a6e90a8 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:21.399420 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1403/2819] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:21 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:21.407086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf296099-78c4-4ad6-9daa-eed39a6e90a8 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:21.414858 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b4905a-4237-4ffc-84d5-546bf3c3e487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:21.415636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b4905a-4237-4ffc-84d5-546bf3c3e487 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:21.415923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b4905a-4237-4ffc-84d5-546bf3c3e487 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:21.416302 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b4905a-4237-4ffc-84d5-546bf3c3e487 admin admin] PATH_INFO: `/v3/users` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.416964 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b4905a-4237-4ffc-84d5-546bf3c3e487 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:21.420232 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b4905a-4237-4ffc-84d5-546bf3c3e487 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:21.454775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1417/2820] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:49:21 2022] POST /identity/v3/users => generated 289 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:21.459482 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a79ebc8-f65d-4371-955d-464b2431418d None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:21.494481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a79ebc8-f65d-4371-955d-464b2431418d None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:21.505203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83335c2a-ee1d-4a99-8dd1-e87a04f95df9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:21.506189 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83335c2a-ee1d-4a99-8dd1-e87a04f95df9 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:21.506694 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83335c2a-ee1d-4a99-8dd1-e87a04f95df9 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:21.507368 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83335c2a-ee1d-4a99-8dd1-e87a04f95df9 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.508557 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83335c2a-ee1d-4a99-8dd1-e87a04f95df9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:21.511836 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83335c2a-ee1d-4a99-8dd1-e87a04f95df9 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:21.534144 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1404/2821] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:21 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:21.538978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b4905a-4237-4ffc-84d5-546bf3c3e487 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:21.554016 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b4905a-4237-4ffc-84d5-546bf3c3e487 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:21.562479 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9489be-61c3-4e3d-81fb-5eac0a652256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:21.563677 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9489be-61c3-4e3d-81fb-5eac0a652256 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:21.564401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9489be-61c3-4e3d-81fb-5eac0a652256 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:21.564722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9489be-61c3-4e3d-81fb-5eac0a652256 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.566024 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9489be-61c3-4e3d-81fb-5eac0a652256 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:21.569284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9489be-61c3-4e3d-81fb-5eac0a652256 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:21.588813 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1418/2822] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:21 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:49:21.594497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83335c2a-ee1d-4a99-8dd1-e87a04f95df9 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:21.614789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83335c2a-ee1d-4a99-8dd1-e87a04f95df9 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:21.625164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:21.626013 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:21.627063 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:21.627564 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] PATH_INFO: `/v3/projects/38542ea2fd62431ab730b12e56194ca5/users/79c5130ea5004c7abe8b558fb40ec17b/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.658764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38542ea2fd62431ab730b12e56194ca5, user_id=79c5130ea5004c7abe8b558fb40ec17b, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:21.661580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:21.683256 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1405/2823] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:49:21 2022] PUT /identity/v3/projects/38542ea2fd62431ab730b12e56194ca5/users/79c5130ea5004c7abe8b558fb40ec17b/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:21.697938 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1e527-dc39-43ff-9c28-d9146966113a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:21.698236 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1e527-dc39-43ff-9c28-d9146966113a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:21.698676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1e527-dc39-43ff-9c28-d9146966113a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:21.752992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-c1e1e527-dc39-43ff-9c28-d9146966113a None None] MFA Rules not processed for user `79c5130ea5004c7abe8b558fb40ec17b`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:21.755358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c1e1e527-dc39-43ff-9c28-d9146966113a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:21.843703 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:21.883433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1419/2824] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:49:22.626644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:22.628738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:22.728024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:22.736747 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77b557c-55b2-4426-b8e9-776206bc8e10 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:22.745164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e7c5a8-1dfd-4cf1-89f6-8af829e4229f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1e61f3604fd4a23ae5f1e7e62ae21ae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:22.746102 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e7c5a8-1dfd-4cf1-89f6-8af829e4229f service glance] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:22.746552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e7c5a8-1dfd-4cf1-89f6-8af829e4229f service glance] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:22.747088 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e7c5a8-1dfd-4cf1-89f6-8af829e4229f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:22.760048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e7c5a8-1dfd-4cf1-89f6-8af829e4229f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:22.761509 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e7c5a8-1dfd-4cf1-89f6-8af829e4229f service glance] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:22.773405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1406/2825] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:22 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:23.083239 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e1e527-dc39-43ff-9c28-d9146966113a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:23.086989 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c1e1e527-dc39-43ff-9c28-d9146966113a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:23.267425 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e1e527-dc39-43ff-9c28-d9146966113a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:23.272641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2528e36-0b05-47ca-b6bc-28f1c85ff8ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:23.273629 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2528e36-0b05-47ca-b6bc-28f1c85ff8ab service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:23.274133 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2528e36-0b05-47ca-b6bc-28f1c85ff8ab service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:23.274639 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2528e36-0b05-47ca-b6bc-28f1c85ff8ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:23.278646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e2528e36-0b05-47ca-b6bc-28f1c85ff8ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:23.296013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2528e36-0b05-47ca-b6bc-28f1c85ff8ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:23.298062 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2528e36-0b05-47ca-b6bc-28f1c85ff8ab service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:23.315350 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1420/2826] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:25.787688 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3584550-dafe-4844-bf1f-dce7b644d41c None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:25.789033 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3584550-dafe-4844-bf1f-dce7b644d41c None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:25.789789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3584550-dafe-4844-bf1f-dce7b644d41c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:25.821238 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f3584550-dafe-4844-bf1f-dce7b644d41c None None] MFA Rules not processed for user `9f7d298884b743f49760437628fc65aa`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:25.823377 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f3584550-dafe-4844-bf1f-dce7b644d41c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:25.909231 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1407/2827] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:28.687542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db26195-6c03-49c8-a7cc-b4a2308872ca None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:28.688423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db26195-6c03-49c8-a7cc-b4a2308872ca None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:28.688423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db26195-6c03-49c8-a7cc-b4a2308872ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:28.732433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5db26195-6c03-49c8-a7cc-b4a2308872ca None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:28.734770 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5db26195-6c03-49c8-a7cc-b4a2308872ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:28.842696 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aec52a-2811-45ea-be2d-32c131e203f6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:28.847795 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aec52a-2811-45ea-be2d-32c131e203f6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:28.847989 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63aec52a-2811-45ea-be2d-32c131e203f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:28.937889 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-63aec52a-2811-45ea-be2d-32c131e203f6 None None] MFA Rules not processed for user `1addff9d7c7348de827c4eba89d41660`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:28.939982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-63aec52a-2811-45ea-be2d-32c131e203f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:28.972215 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1421/2828] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:49:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:49:28.997683 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db26195-6c03-49c8-a7cc-b4a2308872ca None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:29.000124 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5db26195-6c03-49c8-a7cc-b4a2308872ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:29.149627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1408/2829] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:28 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:29.203212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db26195-6c03-49c8-a7cc-b4a2308872ca None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:29.213672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb16931c-eaaf-4cf7-9cc8-bf15738d2a27 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:29.214785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16931c-eaaf-4cf7-9cc8-bf15738d2a27 service swift] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:29.217734 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16931c-eaaf-4cf7-9cc8-bf15738d2a27 service swift] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:29.218665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16931c-eaaf-4cf7-9cc8-bf15738d2a27 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:29.235837 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb16931c-eaaf-4cf7-9cc8-bf15738d2a27 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:29.243513 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb16931c-eaaf-4cf7-9cc8-bf15738d2a27 service swift] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:29.264920 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1422/2830] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:49:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:49:39.125040 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=54 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:39.256299 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:40.825405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63aec52a-2811-45ea-be2d-32c131e203f6 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:40.945588 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63aec52a-2811-45ea-be2d-32c131e203f6 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:40.952930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:40.954146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:40.954944 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:40.956137 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] PATH_INFO: `/v3/users/79c5130ea5004c7abe8b558fb40ec17b` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:40.958433 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] RBAC: Authorizing `identity:delete_user(user_id=79c5130ea5004c7abe8b558fb40ec17b)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:40.961010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:41.124858 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79c5130ea5004c7abe8b558fb40ec17b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:41.126466 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79c5130ea5004c7abe8b558fb40ec17b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:41.144483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79c5130ea5004c7abe8b558fb40ec17b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:41.158116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79c5130ea5004c7abe8b558fb40ec17b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:41.187342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79c5130ea5004c7abe8b558fb40ec17b'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:41.198197 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1409/2831] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:49:40 2022] DELETE /identity/v3/users/79c5130ea5004c7abe8b558fb40ec17b => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:41.205308 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aa77d0-5905-46f2-86f7-e21a45e34db9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:41.205926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aa77d0-5905-46f2-86f7-e21a45e34db9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:41.206277 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aa77d0-5905-46f2-86f7-e21a45e34db9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:41.244488 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f5aa77d0-5905-46f2-86f7-e21a45e34db9 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:41.245775 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f5aa77d0-5905-46f2-86f7-e21a45e34db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:41.398258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1423/2832] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:41.430537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:41.435130 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:41.584797 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e0a1c2-8c62-49c7-8b6d-1bd27190f647 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:41.592022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fac467e-ad7f-4fa0-8bfe-70448be2dd28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:41.593499 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fac467e-ad7f-4fa0-8bfe-70448be2dd28 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:41.594376 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fac467e-ad7f-4fa0-8bfe-70448be2dd28 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:41.595129 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fac467e-ad7f-4fa0-8bfe-70448be2dd28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:41.607110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fac467e-ad7f-4fa0-8bfe-70448be2dd28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:41.608898 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fac467e-ad7f-4fa0-8bfe-70448be2dd28 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:41.625739 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1410/2833] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:42.172521 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aa77d0-5905-46f2-86f7-e21a45e34db9 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:42.175148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f5aa77d0-5905-46f2-86f7-e21a45e34db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:42.199100 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aa77d0-5905-46f2-86f7-e21a45e34db9 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:42.206274 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:42.207044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.207331 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.207605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] PATH_INFO: `/v3/projects/38542ea2fd62431ab730b12e56194ca5` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:42.209397 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] RBAC: Authorizing `identity:delete_project(project_id=38542ea2fd62431ab730b12e56194ca5)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:42.211386 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:42.276378 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38542ea2fd62431ab730b12e56194ca5', 'request_id': 'req-2359932e-9934-4440-bea3-c5118a8d192f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:42.285120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38542ea2fd62431ab730b12e56194ca5', 'request_id': 'req-2359932e-9934-4440-bea3-c5118a8d192f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:42.295266 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38542ea2fd62431ab730b12e56194ca5', 'request_id': 'req-2359932e-9934-4440-bea3-c5118a8d192f'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:42.310582 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] The token cache is being invalidate because project 38542ea2fd62431ab730b12e56194ca5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:49:42.310957 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:42.311626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2359932e-9934-4440-bea3-c5118a8d192f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:42.313117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1424/2834] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:49:42 2022] DELETE /identity/v3/projects/38542ea2fd62431ab730b12e56194ca5 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:42.324244 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54642cf-b88c-42df-9be9-80f770600c57 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.325026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54642cf-b88c-42df-9be9-80f770600c57 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.325026 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54642cf-b88c-42df-9be9-80f770600c57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:42.352410 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c54642cf-b88c-42df-9be9-80f770600c57 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:42.353620 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c54642cf-b88c-42df-9be9-80f770600c57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:42.384827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:42.456333 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1411/2835] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:49:42.466815 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffcd40d-fdcc-499a-ae7e-9358fad6daa5 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.469002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffcd40d-fdcc-499a-ae7e-9358fad6daa5 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.470820 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffcd40d-fdcc-499a-ae7e-9358fad6daa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:42.492245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-9ffcd40d-fdcc-499a-ae7e-9358fad6daa5 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:42.493481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-9ffcd40d-fdcc-499a-ae7e-9358fad6daa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:42.506510 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1425/2836] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:42.511250 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54642cf-b88c-42df-9be9-80f770600c57 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:42.522890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54642cf-b88c-42df-9be9-80f770600c57 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:42.529188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:42.529802 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.530076 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.530342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:42.531031 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:49:42.531331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:42.532934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:42.549155 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1412/2837] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:49:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:49:42.553646 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffcd40d-fdcc-499a-ae7e-9358fad6daa5 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:42.564515 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffcd40d-fdcc-499a-ae7e-9358fad6daa5 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:42.569934 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd9dad5-7f61-4c98-8c7d-79dade0ce0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:42.570828 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd9dad5-7f61-4c98-8c7d-79dade0ce0b1 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.571272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd9dad5-7f61-4c98-8c7d-79dade0ce0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.571759 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd9dad5-7f61-4c98-8c7d-79dade0ce0b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:42.572902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd9dad5-7f61-4c98-8c7d-79dade0ce0b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:42.575830 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd9dad5-7f61-4c98-8c7d-79dade0ce0b1 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:42.623061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1426/2838] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:49:42 2022] POST /identity/v3/projects => generated 383 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:42.632570 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:42.643245 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b1c0f9-7def-4500-8607-597ec54f0efa admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:42.737407 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c66e355-8d2d-43d6-9db0-83efd5c8b456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:42.738370 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66e355-8d2d-43d6-9db0-83efd5c8b456 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.738650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66e355-8d2d-43d6-9db0-83efd5c8b456 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.739000 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c66e355-8d2d-43d6-9db0-83efd5c8b456 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:42.739596 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c66e355-8d2d-43d6-9db0-83efd5c8b456 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:42.741579 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c66e355-8d2d-43d6-9db0-83efd5c8b456 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:42.782799 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1413/2839] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:49:42 2022] POST /identity/v3/users => generated 292 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:42.785622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd9dad5-7f61-4c98-8c7d-79dade0ce0b1 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:42.814635 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd9dad5-7f61-4c98-8c7d-79dade0ce0b1 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:42.835430 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1db595-a53b-4f10-922f-201cc8aba685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:42.836847 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1db595-a53b-4f10-922f-201cc8aba685 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.837467 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1db595-a53b-4f10-922f-201cc8aba685 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.838298 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1db595-a53b-4f10-922f-201cc8aba685 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:42.840104 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1db595-a53b-4f10-922f-201cc8aba685 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:42.842602 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1db595-a53b-4f10-922f-201cc8aba685 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:42.879566 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c66e355-8d2d-43d6-9db0-83efd5c8b456 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:42.883261 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1427/2840] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:42 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:42.893066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c66e355-8d2d-43d6-9db0-83efd5c8b456 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:42.899790 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3ef1d5-aaec-4743-aec8-070ffc93757e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:42.900658 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3ef1d5-aaec-4743-aec8-070ffc93757e admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.900925 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3ef1d5-aaec-4743-aec8-070ffc93757e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.901198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3ef1d5-aaec-4743-aec8-070ffc93757e admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:42.901914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3ef1d5-aaec-4743-aec8-070ffc93757e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:42.903625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3ef1d5-aaec-4743-aec8-070ffc93757e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:42.922022 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1414/2841] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:42 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:42.927744 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1db595-a53b-4f10-922f-201cc8aba685 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:42.947229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1db595-a53b-4f10-922f-201cc8aba685 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:42.958042 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:42.959243 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:42.959739 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:42.960295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] PATH_INFO: `/v3/projects/6ce2dbfdb61842bba27098b7cb813534/users/9aa494ea941646fa94fbe1bb7a849ed1/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:43.015834 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ce2dbfdb61842bba27098b7cb813534, user_id=9aa494ea941646fa94fbe1bb7a849ed1, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:43.031542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:43.069007 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1428/2842] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:49:42 2022] PUT /identity/v3/projects/6ce2dbfdb61842bba27098b7cb813534/users/9aa494ea941646fa94fbe1bb7a849ed1/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:43.087912 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a43116-4772-4880-85d3-1be09d62930e None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:43.088386 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a43116-4772-4880-85d3-1be09d62930e None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:43.088884 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a43116-4772-4880-85d3-1be09d62930e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:43.128203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-22a43116-4772-4880-85d3-1be09d62930e None None] MFA Rules not processed for user `9aa494ea941646fa94fbe1bb7a849ed1`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:43.129819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-22a43116-4772-4880-85d3-1be09d62930e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:43.252634 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1415/2843] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:49:43.269436 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:43.271658 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:43.310267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:43.474477 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e397943a-66f9-4f4f-a893-dee7bb7ea34f admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:43.479526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8eb59c8-aa15-4652-9411-269865d6fde2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:43.480288 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8eb59c8-aa15-4652-9411-269865d6fde2 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:43.484174 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8eb59c8-aa15-4652-9411-269865d6fde2 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:43.485144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8eb59c8-aa15-4652-9411-269865d6fde2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:43.520182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8eb59c8-aa15-4652-9411-269865d6fde2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:43.523138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8eb59c8-aa15-4652-9411-269865d6fde2 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:43.547443 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1429/2844] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:49:51.636934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a43116-4772-4880-85d3-1be09d62930e None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:51.664125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:51.742523 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a43116-4772-4880-85d3-1be09d62930e None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:51.748876 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:51.749715 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:51.750204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:51.750689 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] PATH_INFO: `/v3/users/9aa494ea941646fa94fbe1bb7a849ed1` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:51.752662 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] RBAC: Authorizing `identity:delete_user(user_id=9aa494ea941646fa94fbe1bb7a849ed1)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:51.756301 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:51.823736 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9aa494ea941646fa94fbe1bb7a849ed1'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:51.824544 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9aa494ea941646fa94fbe1bb7a849ed1'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:51.835631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9aa494ea941646fa94fbe1bb7a849ed1'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:51.850150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9aa494ea941646fa94fbe1bb7a849ed1'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:51.868204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9aa494ea941646fa94fbe1bb7a849ed1'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:51.879030 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1416/2845] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:49:51 2022] DELETE /identity/v3/users/9aa494ea941646fa94fbe1bb7a849ed1 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:49:51.885138 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9a817f-7a55-4c28-8886-bb9cab05a196 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:51.885974 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9a817f-7a55-4c28-8886-bb9cab05a196 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:51.886156 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9a817f-7a55-4c28-8886-bb9cab05a196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:51.912026 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-2c9a817f-7a55-4c28-8886-bb9cab05a196 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:51.913312 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2c9a817f-7a55-4c28-8886-bb9cab05a196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:52.004212 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1430/2846] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:52.019213 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:52.021926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:52.121516 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4ba2d3-a5bb-4d6b-8278-402a2055582c admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:52.128448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bef70d-9932-4827-be1e-51acd96ebce2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:52.128705 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bef70d-9932-4827-be1e-51acd96ebce2 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:52.129125 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bef70d-9932-4827-be1e-51acd96ebce2 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:52.129627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bef70d-9932-4827-be1e-51acd96ebce2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:52.146109 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bef70d-9932-4827-be1e-51acd96ebce2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:52.148841 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bef70d-9932-4827-be1e-51acd96ebce2 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:52.167018 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1417/2847] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:52.492325 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9a817f-7a55-4c28-8886-bb9cab05a196 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:52.494812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-2c9a817f-7a55-4c28-8886-bb9cab05a196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:52.525229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9a817f-7a55-4c28-8886-bb9cab05a196 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:52.534682 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:52.535361 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:52.535643 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:52.535945 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] PATH_INFO: `/v3/projects/6ce2dbfdb61842bba27098b7cb813534` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:52.537109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ce2dbfdb61842bba27098b7cb813534)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:52.539117 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:52.606953 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ce2dbfdb61842bba27098b7cb813534', 'request_id': 'req-178560e6-802e-4cda-848a-9d5eb151d26b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:52.613800 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ce2dbfdb61842bba27098b7cb813534', 'request_id': 'req-178560e6-802e-4cda-848a-9d5eb151d26b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:52.620037 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ce2dbfdb61842bba27098b7cb813534', 'request_id': 'req-178560e6-802e-4cda-848a-9d5eb151d26b'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:52.637899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] The token cache is being invalidate because project 6ce2dbfdb61842bba27098b7cb813534 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:49:52.638237 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:52.639044 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-178560e6-802e-4cda-848a-9d5eb151d26b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:52.640805 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1431/2848] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:49:52 2022] DELETE /identity/v3/projects/6ce2dbfdb61842bba27098b7cb813534 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:52.645869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e461268e-76b3-4cf6-a421-2f790e435e7b None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:52.646415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e461268e-76b3-4cf6-a421-2f790e435e7b None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:52.646415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e461268e-76b3-4cf6-a421-2f790e435e7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:52.671923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-e461268e-76b3-4cf6-a421-2f790e435e7b None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:52.673053 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e461268e-76b3-4cf6-a421-2f790e435e7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:52.711877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:52.749584 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1418/2849] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:52.757992 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e23f7f0-85a3-4caf-b8b1-8bb28a2792a0 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:52.758286 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e23f7f0-85a3-4caf-b8b1-8bb28a2792a0 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:52.758568 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e23f7f0-85a3-4caf-b8b1-8bb28a2792a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:52.787314 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-1e23f7f0-85a3-4caf-b8b1-8bb28a2792a0 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:52.789581 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-1e23f7f0-85a3-4caf-b8b1-8bb28a2792a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:52.801143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1432/2850] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:52.805998 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e461268e-76b3-4cf6-a421-2f790e435e7b None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:52.821672 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e461268e-76b3-4cf6-a421-2f790e435e7b None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:52.826830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:52.827396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:52.827670 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:52.828204 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:52.828549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:49:52.828885 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:52.830492 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:52.850679 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1419/2851] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:49:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:49:52.856189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e23f7f0-85a3-4caf-b8b1-8bb28a2792a0 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:52.866245 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e23f7f0-85a3-4caf-b8b1-8bb28a2792a0 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:52.871167 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e4251-8021-4859-a182-ac936b9c9e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:52.871867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e4251-8021-4859-a182-ac936b9c9e7c admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:52.872203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e4251-8021-4859-a182-ac936b9c9e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:52.872512 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e4251-8021-4859-a182-ac936b9c9e7c admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:52.873193 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612e4251-8021-4859-a182-ac936b9c9e7c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:52.875433 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612e4251-8021-4859-a182-ac936b9c9e7c admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:52.896514 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:52.911526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1433/2852] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:49:52 2022] POST /identity/v3/projects => generated 385 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:52.919156 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:52.932241 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c0c076-3ae5-4e6f-8c8f-660bad418751 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:53.029347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b071a111-df9f-403b-a4b0-1597fc35fa99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:53.030396 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b071a111-df9f-403b-a4b0-1597fc35fa99 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:53.030888 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b071a111-df9f-403b-a4b0-1597fc35fa99 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:53.031379 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b071a111-df9f-403b-a4b0-1597fc35fa99 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:53.032447 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b071a111-df9f-403b-a4b0-1597fc35fa99 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:53.035848 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b071a111-df9f-403b-a4b0-1597fc35fa99 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:53.062127 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1420/2853] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:49:52 2022] POST /identity/v3/users => generated 293 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:49:53.073638 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e4251-8021-4859-a182-ac936b9c9e7c admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:53.092321 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e4251-8021-4859-a182-ac936b9c9e7c admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:53.100845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40591db1-ee06-4352-b1d1-fb9f0122ac48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:53.101916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40591db1-ee06-4352-b1d1-fb9f0122ac48 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:53.102411 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40591db1-ee06-4352-b1d1-fb9f0122ac48 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:53.102931 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40591db1-ee06-4352-b1d1-fb9f0122ac48 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:53.104085 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40591db1-ee06-4352-b1d1-fb9f0122ac48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:53.107811 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40591db1-ee06-4352-b1d1-fb9f0122ac48 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:53.122072 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1434/2854] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:53 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:53.138263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b071a111-df9f-403b-a4b0-1597fc35fa99 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:53.150124 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b071a111-df9f-403b-a4b0-1597fc35fa99 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:53.157982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c0a359-d2bd-4da7-8042-8b60cf0567c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:53.158868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c0a359-d2bd-4da7-8042-8b60cf0567c8 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:53.159359 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c0a359-d2bd-4da7-8042-8b60cf0567c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:53.159780 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c0a359-d2bd-4da7-8042-8b60cf0567c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:53.160768 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c0a359-d2bd-4da7-8042-8b60cf0567c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:53.162402 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c0a359-d2bd-4da7-8042-8b60cf0567c8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:53.185362 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1421/2855] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:53 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:53.186548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40591db1-ee06-4352-b1d1-fb9f0122ac48 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:53.205169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40591db1-ee06-4352-b1d1-fb9f0122ac48 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:53.212885 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:53.213862 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:53.214235 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:53.214721 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] PATH_INFO: `/v3/projects/19153ae337624407bb3b806619abe39d/users/c4b6024116b245e78aa557c370265f51/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:53.237013 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19153ae337624407bb3b806619abe39d, user_id=c4b6024116b245e78aa557c370265f51, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:53.239722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:53.262659 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1435/2856] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:49:53 2022] PUT /identity/v3/projects/19153ae337624407bb3b806619abe39d/users/c4b6024116b245e78aa557c370265f51/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:53.279183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2b01ef-589c-4fb3-85fe-f48c5f1ee6cc None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:53.279646 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2b01ef-589c-4fb3-85fe-f48c5f1ee6cc None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:53.280082 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2b01ef-589c-4fb3-85fe-f48c5f1ee6cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:53.328505 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1a2b01ef-589c-4fb3-85fe-f48c5f1ee6cc None None] MFA Rules not processed for user `c4b6024116b245e78aa557c370265f51`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:53.330045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1a2b01ef-589c-4fb3-85fe-f48c5f1ee6cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:53.408667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1422/2857] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:53.420894 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:53.422726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:53.491653 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b499acc2-accf-4d9c-a940-84340451f601 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:53.495509 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:53.496311 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:53.496654 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:53.496926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:53.507301 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:53.508750 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:53.518751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1436/2858] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:54.464791 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1930255d-3679-42a3-b707-69688ed3e4a6 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:54.465495 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1930255d-3679-42a3-b707-69688ed3e4a6 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:54.466010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1930255d-3679-42a3-b707-69688ed3e4a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:54.489106 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-1930255d-3679-42a3-b707-69688ed3e4a6 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:54.490336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-1930255d-3679-42a3-b707-69688ed3e4a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:54.539999 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:54.571539 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1423/2859] 10.222.0.77 () {58 vars in 1009 bytes} [Mon Aug 29 22:49:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:54.589637 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:54.590152 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:54.665216 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70de10da-ca71-4f30-a9d4-81616054ec05 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:54.671481 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e282381-8bf0-4813-8db5-7e645900c2fe service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:54.672114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e282381-8bf0-4813-8db5-7e645900c2fe service swift] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:54.672393 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e282381-8bf0-4813-8db5-7e645900c2fe service swift] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:54.672722 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e282381-8bf0-4813-8db5-7e645900c2fe service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:54.682098 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e282381-8bf0-4813-8db5-7e645900c2fe service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:54.684395 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e282381-8bf0-4813-8db5-7e645900c2fe service swift] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:54.693284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1437/2860] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:49:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:49:56.395263 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1930255d-3679-42a3-b707-69688ed3e4a6 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:56.499338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1930255d-3679-42a3-b707-69688ed3e4a6 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:56.504326 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:56.505423 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:56.505851 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:56.506148 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] PATH_INFO: `/v3/users/c4b6024116b245e78aa557c370265f51` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:56.507528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4b6024116b245e78aa557c370265f51)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:56.509470 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:56.612461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4b6024116b245e78aa557c370265f51'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:56.613200 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4b6024116b245e78aa557c370265f51'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:56.639890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4b6024116b245e78aa557c370265f51'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:56.656075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4b6024116b245e78aa557c370265f51'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:56.683269 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4b6024116b245e78aa557c370265f51'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:56.705259 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1424/2861] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:49:56 2022] DELETE /identity/v3/users/c4b6024116b245e78aa557c370265f51 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:56.710812 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9d0264-984c-4769-97ca-3fd541124859 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:56.711471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9d0264-984c-4769-97ca-3fd541124859 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:56.711471 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9d0264-984c-4769-97ca-3fd541124859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:56.768963 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-5e9d0264-984c-4769-97ca-3fd541124859 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:56.770884 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5e9d0264-984c-4769-97ca-3fd541124859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:56.852143 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:56.868295 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1438/2862] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:56.879986 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:56.881623 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:57.000415 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398ba0-52f6-42e7-95e9-fae6afbf9996 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:57.005819 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b944d2c0-a587-4d01-8424-6a2e0bcc6dfa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:57.007005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944d2c0-a587-4d01-8424-6a2e0bcc6dfa service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:57.008749 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944d2c0-a587-4d01-8424-6a2e0bcc6dfa service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:57.009507 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944d2c0-a587-4d01-8424-6a2e0bcc6dfa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:57.021626 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b944d2c0-a587-4d01-8424-6a2e0bcc6dfa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:57.023627 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b944d2c0-a587-4d01-8424-6a2e0bcc6dfa service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:57.044203 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1425/2863] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:49:57.640552 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9d0264-984c-4769-97ca-3fd541124859 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:57.642641 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-5e9d0264-984c-4769-97ca-3fd541124859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:57.683690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9d0264-984c-4769-97ca-3fd541124859 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:57.708609 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:57.709363 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:57.709642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:57.709911 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] PATH_INFO: `/v3/projects/19153ae337624407bb3b806619abe39d` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:57.711233 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19153ae337624407bb3b806619abe39d)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:57.713246 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:57.801979 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19153ae337624407bb3b806619abe39d', 'request_id': 'req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:57.812129 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19153ae337624407bb3b806619abe39d', 'request_id': 'req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:57.831071 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19153ae337624407bb3b806619abe39d', 'request_id': 'req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:57.852853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] The token cache is being invalidate because project 19153ae337624407bb3b806619abe39d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:49:57.852853 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:57.854549 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-2cef0928-ccab-41c4-9a2c-1c55ba08ac20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:49:57.857032 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1439/2864] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:49:57 2022] DELETE /identity/v3/projects/19153ae337624407bb3b806619abe39d => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:57.862809 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f9f71f-19c7-43fb-8a80-d78958ecad03 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:57.863595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f9f71f-19c7-43fb-8a80-d78958ecad03 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:57.863595 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f9f71f-19c7-43fb-8a80-d78958ecad03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:57.873226 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:57.891510 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-a5f9f71f-19c7-43fb-8a80-d78958ecad03 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:57.893419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-a5f9f71f-19c7-43fb-8a80-d78958ecad03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:57.986985 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1426/2865] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:57.997053 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a0765-a8f3-4a50-a548-fadbb786db74 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:57.997536 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a0765-a8f3-4a50-a548-fadbb786db74 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:57.997970 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a0765-a8f3-4a50-a548-fadbb786db74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:58.044651 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-b41a0765-a8f3-4a50-a548-fadbb786db74 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:58.047500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-b41a0765-a8f3-4a50-a548-fadbb786db74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:58.062804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1440/2866] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:49:58.068871 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f9f71f-19c7-43fb-8a80-d78958ecad03 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:58.080392 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f9f71f-19c7-43fb-8a80-d78958ecad03 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:58.085934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:58.086580 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:58.086881 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:58.087160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:58.087840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:49:58.088146 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:58.089867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:58.104092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1427/2867] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:49:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 29 22:49:58.109848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41a0765-a8f3-4a50-a548-fadbb786db74 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:58.121462 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41a0765-a8f3-4a50-a548-fadbb786db74 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:58.127022 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada3249f-bbf7-4a04-8238-8cab268cf9a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:58.127746 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada3249f-bbf7-4a04-8238-8cab268cf9a3 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:58.128097 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada3249f-bbf7-4a04-8238-8cab268cf9a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:58.128532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada3249f-bbf7-4a04-8238-8cab268cf9a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:58.129668 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada3249f-bbf7-4a04-8238-8cab268cf9a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:58.132228 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada3249f-bbf7-4a04-8238-8cab268cf9a3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:58.187105 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1441/2868] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:49:58 2022] POST /identity/v3/projects => generated 399 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 29 22:49:58.188571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:58.205380 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0f8ca4-fb52-447f-a374-e1d66c9f4799 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:58.287222 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafa8b6-e6c0-44cd-a828-e02edf3a9812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:58.288188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafa8b6-e6c0-44cd-a828-e02edf3a9812 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:58.288577 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafa8b6-e6c0-44cd-a828-e02edf3a9812 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:58.288947 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafa8b6-e6c0-44cd-a828-e02edf3a9812 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:58.289687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edafa8b6-e6c0-44cd-a828-e02edf3a9812 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:58.291868 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edafa8b6-e6c0-44cd-a828-e02edf3a9812 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:58.323195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1428/2869] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:49:58 2022] POST /identity/v3/users => generated 300 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:49:58.326086 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada3249f-bbf7-4a04-8238-8cab268cf9a3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:58.343416 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada3249f-bbf7-4a04-8238-8cab268cf9a3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:58.348874 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f9737-ec52-4729-8206-aff9910c6ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:58.349502 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f9737-ec52-4729-8206-aff9910c6ee4 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:58.349781 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f9737-ec52-4729-8206-aff9910c6ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:58.350054 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f9737-ec52-4729-8206-aff9910c6ee4 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:58.350718 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6f9737-ec52-4729-8206-aff9910c6ee4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:58.352381 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6f9737-ec52-4729-8206-aff9910c6ee4 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:58.373398 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1442/2870] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 29 22:49:58.402878 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafa8b6-e6c0-44cd-a828-e02edf3a9812 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:58.431419 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafa8b6-e6c0-44cd-a828-e02edf3a9812 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:58.437930 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a7e945-0efe-4944-b5dd-9f3e0e868242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:58.439183 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7e945-0efe-4944-b5dd-9f3e0e868242 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:58.439640 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7e945-0efe-4944-b5dd-9f3e0e868242 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:58.440068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7e945-0efe-4944-b5dd-9f3e0e868242 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:58.440903 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a7e945-0efe-4944-b5dd-9f3e0e868242 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:58.442697 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a7e945-0efe-4944-b5dd-9f3e0e868242 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:58.456695 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1429/2871] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:49:58.471532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f9737-ec52-4729-8206-aff9910c6ee4 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:58.511910 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f9737-ec52-4729-8206-aff9910c6ee4 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:58.523524 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:58.524538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:58.524816 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:58.525220 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] PATH_INFO: `/v3/projects/8b1aaadf54034bb68b2bd84fd65befb6/users/6f20fa648ac0420782798513399cbf6e/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:58.563128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b1aaadf54034bb68b2bd84fd65befb6, user_id=6f20fa648ac0420782798513399cbf6e, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:58.570181 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:58.618077 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1443/2872] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:49:58 2022] PUT /identity/v3/projects/8b1aaadf54034bb68b2bd84fd65befb6/users/6f20fa648ac0420782798513399cbf6e/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:49:58.638397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c0f9b-b0e7-459a-a8f4-5f5805c0939f None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:58.638957 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c0f9b-b0e7-459a-a8f4-5f5805c0939f None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:58.639358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c0f9b-b0e7-459a-a8f4-5f5805c0939f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:58.696893 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-364c0f9b-b0e7-459a-a8f4-5f5805c0939f None None] MFA Rules not processed for user `6f20fa648ac0420782798513399cbf6e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:49:58.699547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-364c0f9b-b0e7-459a-a8f4-5f5805c0939f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:58.874610 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:49:58.913343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1430/2873] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:49:58 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:49:58.929315 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:49:58.933330 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:49:59.072869 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18217ace-8791-4927-b83e-6a04f36b900b admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:49:59.078888 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d763c92d-fbab-4077-8498-aa82d7abb893 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7964b4109f9a4ac6a68d90366489557c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:49:59.079714 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d763c92d-fbab-4077-8498-aa82d7abb893 service cinder] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:49:59.080123 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d763c92d-fbab-4077-8498-aa82d7abb893 service cinder] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:49:59.080503 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d763c92d-fbab-4077-8498-aa82d7abb893 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:49:59.103132 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d763c92d-fbab-4077-8498-aa82d7abb893 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:49:59.105972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d763c92d-fbab-4077-8498-aa82d7abb893 service cinder] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:49:59.124833 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1444/2874] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:49:58 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:50:01.580784 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364c0f9b-b0e7-459a-a8f4-5f5805c0939f None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:01.582375 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-364c0f9b-b0e7-459a-a8f4-5f5805c0939f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:01.675782 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364c0f9b-b0e7-459a-a8f4-5f5805c0939f None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:01.682049 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:01.683896 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:01.684291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:01.684678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:01.686599 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:01.799343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:01.800924 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:01.817506 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1431/2875] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:50:01 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:50:08.238917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d763c92d-fbab-4077-8498-aa82d7abb893 service cinder] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:08.255356 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d763c92d-fbab-4077-8498-aa82d7abb893 service cinder] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:08.261814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e34df-10b5-40fe-a239-242dfdab68bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:08.262902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e34df-10b5-40fe-a239-242dfdab68bd service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:08.265559 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e34df-10b5-40fe-a239-242dfdab68bd service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:08.265976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e34df-10b5-40fe-a239-242dfdab68bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:08.268925 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-d80e34df-10b5-40fe-a239-242dfdab68bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:08.284227 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:08.295607 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:08.391439 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80e34df-10b5-40fe-a239-242dfdab68bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:08.395431 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80e34df-10b5-40fe-a239-242dfdab68bd service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:08.407400 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a253dd-950c-4af0-95be-838c661e00a4 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:08.415451 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:08.416358 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:08.417441 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:08.418196 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] PATH_INFO: `/v3/users/6f20fa648ac0420782798513399cbf6e` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:08.419859 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f20fa648ac0420782798513399cbf6e)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:08.422446 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:08.435300 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1445/2876] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:50:08 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:50:08.528113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f20fa648ac0420782798513399cbf6e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:08.529835 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f20fa648ac0420782798513399cbf6e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:08.541172 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:08.560445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f20fa648ac0420782798513399cbf6e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:08.575126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f20fa648ac0420782798513399cbf6e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:08.611485 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f20fa648ac0420782798513399cbf6e'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:08.626651 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1432/2877] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:50:08 2022] DELETE /identity/v3/users/6f20fa648ac0420782798513399cbf6e => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:50:08.629164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e5cc7-5ce3-4594-ac84-6aeef17c061a None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:08.629686 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e5cc7-5ce3-4594-ac84-6aeef17c061a None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:08.630182 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e5cc7-5ce3-4594-ac84-6aeef17c061a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:08.661275 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-de4e5cc7-5ce3-4594-ac84-6aeef17c061a None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:50:08.663336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-de4e5cc7-5ce3-4594-ac84-6aeef17c061a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:08.896490 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1446/2878] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:50:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:50:08.918056 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:08.920169 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:09.061071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f85fccd-5d47-444a-b1b1-2a587b9302f4 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:09.067818 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ed8c4c-d039-47fe-817a-46bd8c44c243 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:09.068883 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ed8c4c-d039-47fe-817a-46bd8c44c243 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:09.069445 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ed8c4c-d039-47fe-817a-46bd8c44c243 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:09.070113 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ed8c4c-d039-47fe-817a-46bd8c44c243 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:09.086337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ed8c4c-d039-47fe-817a-46bd8c44c243 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:09.089644 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ed8c4c-d039-47fe-817a-46bd8c44c243 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:09.112727 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1433/2879] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:50:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:50:09.707517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4e5cc7-5ce3-4594-ac84-6aeef17c061a None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:09.709434 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-de4e5cc7-5ce3-4594-ac84-6aeef17c061a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:09.722505 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4e5cc7-5ce3-4594-ac84-6aeef17c061a None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:09.727310 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:09.727948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:09.728226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:09.729144 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] PATH_INFO: `/v3/projects/8b1aaadf54034bb68b2bd84fd65befb6` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:09.730861 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b1aaadf54034bb68b2bd84fd65befb6)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:09.732780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:09.820011 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b1aaadf54034bb68b2bd84fd65befb6', 'request_id': 'req-c742df6a-b958-4931-aabd-ca0305bc583d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:09.832345 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b1aaadf54034bb68b2bd84fd65befb6', 'request_id': 'req-c742df6a-b958-4931-aabd-ca0305bc583d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:09.847768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b1aaadf54034bb68b2bd84fd65befb6', 'request_id': 'req-c742df6a-b958-4931-aabd-ca0305bc583d'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:09.866078 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] The token cache is being invalidate because project 8b1aaadf54034bb68b2bd84fd65befb6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:50:09.866530 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:09.867292 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:09.869939 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1447/2880] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:50:09 2022] DELETE /identity/v3/projects/8b1aaadf54034bb68b2bd84fd65befb6 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:50:15.865288 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81a0049-4786-48ff-a5c9-d240f6a90abf None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:15.865838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81a0049-4786-48ff-a5c9-d240f6a90abf None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:15.865838 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81a0049-4786-48ff-a5c9-d240f6a90abf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:15.892645 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-f81a0049-4786-48ff-a5c9-d240f6a90abf None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:50:15.894709 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f81a0049-4786-48ff-a5c9-d240f6a90abf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:15.956103 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1434/2881] 10.222.0.77 () {58 vars in 967 bytes} [Mon Aug 29 22:50:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:50:15.971179 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:15.972932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:16.017917 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:16.051832 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c742df6a-b958-4931-aabd-ca0305bc583d admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:16.057227 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e1df04-c882-40d3-8573-85fbb145b142 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:16.058365 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e1df04-c882-40d3-8573-85fbb145b142 service swift] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:16.058623 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e1df04-c882-40d3-8573-85fbb145b142 service swift] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:16.058928 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e1df04-c882-40d3-8573-85fbb145b142 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:16.067450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e1df04-c882-40d3-8573-85fbb145b142 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:16.069079 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e1df04-c882-40d3-8573-85fbb145b142 service swift] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:16.078428 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1448/2882] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:50:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 29 22:50:18.149461 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81a0049-4786-48ff-a5c9-d240f6a90abf None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:18.152562 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f81a0049-4786-48ff-a5c9-d240f6a90abf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:18.246432 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81a0049-4786-48ff-a5c9-d240f6a90abf None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:18.257150 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:18.257794 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:18.258075 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:18.258348 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] PATH_INFO: `/v3/users/17a5395280b342de9a39f2a984a2c0ba` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:18.259573 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] RBAC: Authorizing `identity:delete_user(user_id=17a5395280b342de9a39f2a984a2c0ba)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:18.261532 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:18.283667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:18.337677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17a5395280b342de9a39f2a984a2c0ba'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.338235 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17a5395280b342de9a39f2a984a2c0ba'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.353167 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17a5395280b342de9a39f2a984a2c0ba'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.363306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17a5395280b342de9a39f2a984a2c0ba'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.380168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-0623e61a-5843-4813-8189-7dbedf6c836a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17a5395280b342de9a39f2a984a2c0ba'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.386962 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1435/2883] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:50:18 2022] DELETE /identity/v3/users/17a5395280b342de9a39f2a984a2c0ba => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:50:18.583448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e1df04-c882-40d3-8573-85fbb145b142 service swift] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:18.597401 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e1df04-c882-40d3-8573-85fbb145b142 service swift] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:18.677500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:18.678150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:18.678427 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:18.678699 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] PATH_INFO: `/v3/projects/b1f240a0b08841edb5906875c5dcf3e7` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:18.679867 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1f240a0b08841edb5906875c5dcf3e7)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:18.681922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:18.735672 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1f240a0b08841edb5906875c5dcf3e7', 'request_id': 'req-afed0341-b19c-47e9-9855-ccbd149dbb42'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.745666 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1f240a0b08841edb5906875c5dcf3e7', 'request_id': 'req-afed0341-b19c-47e9-9855-ccbd149dbb42'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.752622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1f240a0b08841edb5906875c5dcf3e7', 'request_id': 'req-afed0341-b19c-47e9-9855-ccbd149dbb42'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.767545 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] The token cache is being invalidate because project b1f240a0b08841edb5906875c5dcf3e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103002) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:50:18.768074 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.769088 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-afed0341-b19c-47e9-9855-ccbd149dbb42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:50:18.770969 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1449/2884] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:50:18 2022] DELETE /identity/v3/projects/b1f240a0b08841edb5906875c5dcf3e7 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:50:18.781491 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de4aa6-1d2a-4ded-b459-a7548ae21c87 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:18.781781 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de4aa6-1d2a-4ded-b459-a7548ae21c87 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:18.782066 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de4aa6-1d2a-4ded-b459-a7548ae21c87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:18.808771 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-c3de4aa6-1d2a-4ded-b459-a7548ae21c87 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:50:18.809994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-c3de4aa6-1d2a-4ded-b459-a7548ae21c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:18.888667 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1436/2885] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:50:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:50:18.895241 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bcd5fd-4df6-445b-af87-9fc6e5e91da4 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:18.895528 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bcd5fd-4df6-445b-af87-9fc6e5e91da4 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:18.895802 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bcd5fd-4df6-445b-af87-9fc6e5e91da4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:18.916442 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-76bcd5fd-4df6-445b-af87-9fc6e5e91da4 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:50:18.917710 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-76bcd5fd-4df6-445b-af87-9fc6e5e91da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:18.926264 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1450/2886] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:50:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:50:18.931025 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3de4aa6-1d2a-4ded-b459-a7548ae21c87 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:18.942243 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3de4aa6-1d2a-4ded-b459-a7548ae21c87 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:18.947239 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:18.947842 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:18.948168 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:18.948964 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] PATH_INFO: `/v3/domains` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:18.949621 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=103003) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 29 22:50:18.949923 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:18.951560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:18.965187 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1437/2887] 10.222.0.77 () {58 vars in 1153 bytes} [Mon Aug 29 22:50:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 29 22:50:18.971455 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bcd5fd-4df6-445b-af87-9fc6e5e91da4 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:18.989448 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bcd5fd-4df6-445b-af87-9fc6e5e91da4 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:18.996284 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522083d3-b6e2-4bb7-b44b-96bc00ee8a11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:18.996912 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522083d3-b6e2-4bb7-b44b-96bc00ee8a11 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:18.997198 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522083d3-b6e2-4bb7-b44b-96bc00ee8a11 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:18.997473 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522083d3-b6e2-4bb7-b44b-96bc00ee8a11 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:18.998134 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522083d3-b6e2-4bb7-b44b-96bc00ee8a11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:19.000092 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522083d3-b6e2-4bb7-b44b-96bc00ee8a11 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:19.039952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1451/2888] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:50:18 2022] POST /identity/v3/projects => generated 371 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:50:19.041094 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:19.052098 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256a4ac7-bdb4-47c3-8c51-61b9e2bfae65 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:19.119023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f85f5b-b412-47b9-a0d8-217cdbc5d82e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:19.119642 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f85f5b-b412-47b9-a0d8-217cdbc5d82e admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:19.120448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f85f5b-b412-47b9-a0d8-217cdbc5d82e admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:19.120892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f85f5b-b412-47b9-a0d8-217cdbc5d82e admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:19.121540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f85f5b-b412-47b9-a0d8-217cdbc5d82e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:19.123511 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f85f5b-b412-47b9-a0d8-217cdbc5d82e admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:19.147383 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1438/2889] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:50:19 2022] POST /identity/v3/users => generated 286 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:50:19.152290 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522083d3-b6e2-4bb7-b44b-96bc00ee8a11 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:19.162841 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522083d3-b6e2-4bb7-b44b-96bc00ee8a11 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:19.167739 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35568ac2-518f-4f9e-ad13-cea73bdcfffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:19.168336 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35568ac2-518f-4f9e-ad13-cea73bdcfffb admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:19.168622 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35568ac2-518f-4f9e-ad13-cea73bdcfffb admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:19.168892 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35568ac2-518f-4f9e-ad13-cea73bdcfffb admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:19.169538 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35568ac2-518f-4f9e-ad13-cea73bdcfffb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:19.171191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35568ac2-518f-4f9e-ad13-cea73bdcfffb admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:19.181619 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1452/2890] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:50:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:50:19.186224 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f85f5b-b412-47b9-a0d8-217cdbc5d82e admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:19.196853 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f85f5b-b412-47b9-a0d8-217cdbc5d82e admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:19.202982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417bc8f2-4eaa-44c6-9b5d-f5a8628896d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:19.203531 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417bc8f2-4eaa-44c6-9b5d-f5a8628896d8 admin admin] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:19.203814 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417bc8f2-4eaa-44c6-9b5d-f5a8628896d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:19.204087 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417bc8f2-4eaa-44c6-9b5d-f5a8628896d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:19.204753 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417bc8f2-4eaa-44c6-9b5d-f5a8628896d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:19.206369 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417bc8f2-4eaa-44c6-9b5d-f5a8628896d8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:19.215755 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:19.222892 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1439/2891] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:50:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:50:19.227127 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35568ac2-518f-4f9e-ad13-cea73bdcfffb admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:19.238307 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35568ac2-518f-4f9e-ad13-cea73bdcfffb admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:19.243055 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:19.243701 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] REQUEST_METHOD: `PUT` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:19.243984 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:19.244247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] PATH_INFO: `/v3/projects/01c16f8389774f4c94889a562cdd38f9/users/7447ba5f53114d278711dccc0aa1ad07/roles/0eb5172435534daf81351f9eb5281719` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:19.258753 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:19.262814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01c16f8389774f4c94889a562cdd38f9, user_id=7447ba5f53114d278711dccc0aa1ad07, role_id=0eb5172435534daf81351f9eb5281719)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:19.266102 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:19.287056 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1453/2892] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:50:19 2022] PUT /identity/v3/projects/01c16f8389774f4c94889a562cdd38f9/users/7447ba5f53114d278711dccc0aa1ad07/roles/0eb5172435534daf81351f9eb5281719 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:50:19.293074 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333a4150-09fa-402d-adb1-7c6d28301158 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:19.293363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333a4150-09fa-402d-adb1-7c6d28301158 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:19.293685 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333a4150-09fa-402d-adb1-7c6d28301158 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:19.315443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-333a4150-09fa-402d-adb1-7c6d28301158 None None] MFA Rules not processed for user `2f1e799f7d5b4182a2e0ecd338aaa415`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:50:19.316668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-333a4150-09fa-402d-adb1-7c6d28301158 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:19.389624 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1440/2893] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:50:19.401046 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:19.402814 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:19.470020 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba458f5-ddd3-4c10-85f9-58206fd00f93 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:19.474642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bd93d7-9077-4a5b-90c0-ce00d7d69269 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:19.474642 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd93d7-9077-4a5b-90c0-ce00d7d69269 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:19.474916 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd93d7-9077-4a5b-90c0-ce00d7d69269 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:19.475094 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd93d7-9077-4a5b-90c0-ce00d7d69269 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:19.488229 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bd93d7-9077-4a5b-90c0-ce00d7d69269 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:19.490959 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bd93d7-9077-4a5b-90c0-ce00d7d69269 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:19.502451 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1454/2894] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:50:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:50:24.366869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bda12b-d516-40db-a9f8-436fb3b801b4 None None] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:24.367372 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bda12b-d516-40db-a9f8-436fb3b801b4 None None] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:24.367669 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bda12b-d516-40db-a9f8-436fb3b801b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:24.412339 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.auth.core [None req-d0bda12b-d516-40db-a9f8-436fb3b801b4 None None] MFA Rules not processed for user `7447ba5f53114d278711dccc0aa1ad07`. Rule list: `[]` (Enabled: `True`). {{(pid=103003) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:50:24.414065 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d0bda12b-d516-40db-a9f8-436fb3b801b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:24.514554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1441/2895] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:50:24 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:50:24.522119 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bd93d7-9077-4a5b-90c0-ce00d7d69269 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:24.535423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bd93d7-9077-4a5b-90c0-ce00d7d69269 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:24.543050 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97fe583-6590-4ab2-820f-0545a1f89191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:24.543690 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97fe583-6590-4ab2-820f-0545a1f89191 admin admin] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:24.543990 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97fe583-6590-4ab2-820f-0545a1f89191 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:24.544258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97fe583-6590-4ab2-820f-0545a1f89191 admin admin] PATH_INFO: `/v3/projects` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:24.544882 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97fe583-6590-4ab2-820f-0545a1f89191 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:24.546978 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97fe583-6590-4ab2-820f-0545a1f89191 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:24.602008 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1455/2896] 10.222.0.77 () {60 vars in 1153 bytes} [Mon Aug 29 22:50:24 2022] POST /identity/v3/projects => generated 373 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:50:24.603657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bda12b-d516-40db-a9f8-436fb3b801b4 None None] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:24.623811 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bda12b-d516-40db-a9f8-436fb3b801b4 None None] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:24.717744 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f3f527-c580-425a-9ae1-fa667c5b72c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:24.718844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f3f527-c580-425a-9ae1-fa667c5b72c6 admin admin] REQUEST_METHOD: `POST` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:24.719133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f3f527-c580-425a-9ae1-fa667c5b72c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:24.719628 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f3f527-c580-425a-9ae1-fa667c5b72c6 admin admin] PATH_INFO: `/v3/users` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:24.720601 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f3f527-c580-425a-9ae1-fa667c5b72c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:24.723877 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f3f527-c580-425a-9ae1-fa667c5b72c6 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:24.767541 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1442/2897] 10.222.0.77 () {60 vars in 1144 bytes} [Mon Aug 29 22:50:24 2022] POST /identity/v3/users => generated 293 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 29 22:50:24.769217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97fe583-6590-4ab2-820f-0545a1f89191 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:24.783485 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97fe583-6590-4ab2-820f-0545a1f89191 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:24.793423 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5a3519-73f9-4198-a73f-b4b361eee4d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:24.794197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5a3519-73f9-4198-a73f-b4b361eee4d3 admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:24.794717 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5a3519-73f9-4198-a73f-b4b361eee4d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:24.795279 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5a3519-73f9-4198-a73f-b4b361eee4d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:24.796061 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5a3519-73f9-4198-a73f-b4b361eee4d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:24.799204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5a3519-73f9-4198-a73f-b4b361eee4d3 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:24.816191 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1456/2898] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:50:24 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:50:24.820619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f3f527-c580-425a-9ae1-fa667c5b72c6 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:24.840210 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f3f527-c580-425a-9ae1-fa667c5b72c6 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:24.856406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae56ca9-9157-4647-8646-2178a5f40bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:24.857674 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae56ca9-9157-4647-8646-2178a5f40bce admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:24.858192 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae56ca9-9157-4647-8646-2178a5f40bce admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:24.858547 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae56ca9-9157-4647-8646-2178a5f40bce admin admin] PATH_INFO: `/v3/projects/8a4dffe025c1425ea038139156586810/users/c2db6d92406a45378a34d8b18b3279cb/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:24.878443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae56ca9-9157-4647-8646-2178a5f40bce admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a4dffe025c1425ea038139156586810, user_id=c2db6d92406a45378a34d8b18b3279cb, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:24.882824 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae56ca9-9157-4647-8646-2178a5f40bce admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:24.887174 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:24.905181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1443/2899] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:50:24 2022] PUT /identity/v3/projects/8a4dffe025c1425ea038139156586810/users/c2db6d92406a45378a34d8b18b3279cb/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:50:24.912029 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5a3519-73f9-4198-a73f-b4b361eee4d3 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:24.928783 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5a3519-73f9-4198-a73f-b4b361eee4d3 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:24.996028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ed7376-4d53-48f0-aa87-652a46f18b1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:24.996644 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed7376-4d53-48f0-aa87-652a46f18b1a admin admin] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:24.996921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed7376-4d53-48f0-aa87-652a46f18b1a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:24.997189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed7376-4d53-48f0-aa87-652a46f18b1a admin admin] PATH_INFO: `/v3/roles` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:24.997849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ed7376-4d53-48f0-aa87-652a46f18b1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:24.999616 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ed7376-4d53-48f0-aa87-652a46f18b1a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:25.012125 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1457/2900] 10.222.0.77 () {58 vars in 1122 bytes} [Mon Aug 29 22:50:24 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 29 22:50:25.016846 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae56ca9-9157-4647-8646-2178a5f40bce admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:25.028248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae56ca9-9157-4647-8646-2178a5f40bce admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:25.033331 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:25.034015 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] REQUEST_METHOD: `PUT` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:25.034295 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:25.034571 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] PATH_INFO: `/v3/projects/8a4dffe025c1425ea038139156586810/users/c2db6d92406a45378a34d8b18b3279cb/roles/38171c89968943bd92d9e9758db65615` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:25.036657 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a4dffe025c1425ea038139156586810, user_id=c2db6d92406a45378a34d8b18b3279cb, role_id=38171c89968943bd92d9e9758db65615)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:25.039394 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:25.054363 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1444/2901] 10.222.0.77 () {60 vars in 1483 bytes} [Mon Aug 29 22:50:25 2022] PUT /identity/v3/projects/8a4dffe025c1425ea038139156586810/users/c2db6d92406a45378a34d8b18b3279cb/roles/38171c89968943bd92d9e9758db65615 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:50:30.571694 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f0354-d724-4f1f-9a4f-cf9156cb24c9 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:30.572558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f0354-d724-4f1f-9a4f-cf9156cb24c9 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:30.573373 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f0354-d724-4f1f-9a4f-cf9156cb24c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:30.610697 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-723f0354-d724-4f1f-9a4f-cf9156cb24c9 None None] MFA Rules not processed for user `c2db6d92406a45378a34d8b18b3279cb`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:50:30.612880 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-723f0354-d724-4f1f-9a4f-cf9156cb24c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:30.693081 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:30.744362 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1458/2902] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:50:30 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:50:30.757537 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:30.759847 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:30.866436 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345d5cd6-acd5-4a50-b2cb-256c5dd77cdd admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:30.873554 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2855e8bf-1c12-4dc7-9946-cf267ed81531 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:30.874188 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2855e8bf-1c12-4dc7-9946-cf267ed81531 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:30.874458 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2855e8bf-1c12-4dc7-9946-cf267ed81531 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:30.874761 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2855e8bf-1c12-4dc7-9946-cf267ed81531 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:30.950274 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2855e8bf-1c12-4dc7-9946-cf267ed81531 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:30.954992 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2855e8bf-1c12-4dc7-9946-cf267ed81531 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:30.972228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1445/2903] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:50:30 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:50:32.238370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befa5b1-35c5-4a56-9ef1-e47168923484 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:32.238370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befa5b1-35c5-4a56-9ef1-e47168923484 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:32.238370 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befa5b1-35c5-4a56-9ef1-e47168923484 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:32.266795 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-3befa5b1-35c5-4a56-9ef1-e47168923484 None None] MFA Rules not processed for user `7447ba5f53114d278711dccc0aa1ad07`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:50:32.268280 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3befa5b1-35c5-4a56-9ef1-e47168923484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:32.277211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1459/2904] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:50:32 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:50:40.965638 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=63,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:42.255999 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=11 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:47.512336 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2855e8bf-1c12-4dc7-9946-cf267ed81531 service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:47.574983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2855e8bf-1c12-4dc7-9946-cf267ed81531 service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:47.578908 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:47.579525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:47.579803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:47.580071 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:47.581857 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:47.597157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:47.598635 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:47.611525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1446/2905] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:50:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:50:48.822306 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3befa5b1-35c5-4a56-9ef1-e47168923484 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:50:48.834231 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3befa5b1-35c5-4a56-9ef1-e47168923484 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:50:48.838866 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:50:48.839500 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:50:48.840038 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:50:48.840324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:50:48.842590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:50:48.858313 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:50:48.860158 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:50:48.873526 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1460/2906] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:50:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:50:57.605855 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=26 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:50:58.867120 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=6 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:51:02.020890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:51:02.031528 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bf77b-b7bc-4aa2-9092-22cdad9fc1a8 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:51:02.035248 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e566f145-da4f-4042-bb88-fc5da81ee0f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:51:02.035850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566f145-da4f-4042-bb88-fc5da81ee0f6 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:02.036126 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566f145-da4f-4042-bb88-fc5da81ee0f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:02.036406 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566f145-da4f-4042-bb88-fc5da81ee0f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:02.046110 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e566f145-da4f-4042-bb88-fc5da81ee0f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:51:02.047589 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e566f145-da4f-4042-bb88-fc5da81ee0f6 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:51:02.057521 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1447/2907] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:51:02 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:51:12.053180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=6 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:51:31.252002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:51:31.267673 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0e458c-4e38-4cc5-88d7-065d9442e988 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:51:31.275940 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0831318-52d7-4d96-8c04-a91be44248f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:51:31.276730 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0831318-52d7-4d96-8c04-a91be44248f7 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:31.277109 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0831318-52d7-4d96-8c04-a91be44248f7 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:31.277491 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0831318-52d7-4d96-8c04-a91be44248f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:31.279741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e0831318-52d7-4d96-8c04-a91be44248f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:51:31.304114 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0831318-52d7-4d96-8c04-a91be44248f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:51:31.308150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0831318-52d7-4d96-8c04-a91be44248f7 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:51:31.328202 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1461/2908] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:51:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:51:31.404133 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e566f145-da4f-4042-bb88-fc5da81ee0f6 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:51:31.416390 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e566f145-da4f-4042-bb88-fc5da81ee0f6 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:51:31.420900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:51:31.427560 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:31.427914 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:31.428724 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:31.432574 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:51:31.446796 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:51:31.448569 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:51:31.462653 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1448/2909] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:51:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:51:39.163339 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4c769-3606-4d73-88d3-78dadfdd89d2 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:39.164204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4c769-3606-4d73-88d3-78dadfdd89d2 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:39.164204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4c769-3606-4d73-88d3-78dadfdd89d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:39.225958 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-61b4c769-3606-4d73-88d3-78dadfdd89d2 None None] MFA Rules not processed for user `18211b2dc99f43769685855f07c64a3e`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:51:39.227776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-61b4c769-3606-4d73-88d3-78dadfdd89d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:51:39.310335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1462/2910] 10.222.0.77 () {58 vars in 967 bytes} [Mon Aug 29 22:51:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:51:39.374609 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:51:39.377273 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:51:39.476434 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27577932-28c1-465c-af44-e5f0f6e93efd service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:51:39.482648 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4617bb94-2826-4962-a61a-4ac9333a9ff1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03361f4b51d34cf28c4b5451c3b1f881', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:51:39.483343 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4617bb94-2826-4962-a61a-4ac9333a9ff1 service swift] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:39.483678 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4617bb94-2826-4962-a61a-4ac9333a9ff1 service swift] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:39.484070 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4617bb94-2826-4962-a61a-4ac9333a9ff1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:39.498448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4617bb94-2826-4962-a61a-4ac9333a9ff1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:51:39.500567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4617bb94-2826-4962-a61a-4ac9333a9ff1 service swift] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:51:39.509905 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1449/2911] 10.222.0.77 () {60 vars in 1384 bytes} [Mon Aug 29 22:51:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 29 22:51:47.001752 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b4c769-3606-4d73-88d3-78dadfdd89d2 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:51:47.004296 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-61b4c769-3606-4d73-88d3-78dadfdd89d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:51:47.098588 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b4c769-3606-4d73-88d3-78dadfdd89d2 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:51:47.106543 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:51:47.107624 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:47.108166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:47.108647 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] PATH_INFO: `/v3/users/9f7d298884b743f49760437628fc65aa` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:47.110271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f7d298884b743f49760437628fc65aa)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:51:47.113716 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:51:47.166267 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f7d298884b743f49760437628fc65aa'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.176113 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f7d298884b743f49760437628fc65aa'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.176947 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f7d298884b743f49760437628fc65aa'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.177614 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f7d298884b743f49760437628fc65aa'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.182360 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:51:47.190541 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f7d298884b743f49760437628fc65aa'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.212558 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1463/2912] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:51:46 2022] DELETE /identity/v3/users/9f7d298884b743f49760437628fc65aa => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:51:47.418929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4617bb94-2826-4962-a61a-4ac9333a9ff1 service swift] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:51:47.437994 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4617bb94-2826-4962-a61a-4ac9333a9ff1 service swift] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:51:47.511677 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:51:47.512314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:47.512594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:47.512866 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] PATH_INFO: `/v3/projects/c814b0ae26d84c08b9c27e1f11d3b2ff` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:47.514014 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c814b0ae26d84c08b9c27e1f11d3b2ff)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:51:47.515970 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:51:47.568340 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c814b0ae26d84c08b9c27e1f11d3b2ff', 'request_id': 'req-aed9471a-4c54-41b5-b165-1cdd186a17c8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.575552 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c814b0ae26d84c08b9c27e1f11d3b2ff', 'request_id': 'req-aed9471a-4c54-41b5-b165-1cdd186a17c8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.579184 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:51:47.583631 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c814b0ae26d84c08b9c27e1f11d3b2ff', 'request_id': 'req-aed9471a-4c54-41b5-b165-1cdd186a17c8'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.596182 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] The token cache is being invalidate because project c814b0ae26d84c08b9c27e1f11d3b2ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:51:47.596481 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.597180 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.598789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1450/2913] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:51:47 2022] DELETE /identity/v3/projects/c814b0ae26d84c08b9c27e1f11d3b2ff => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:51:47.602961 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:51:47.604459 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:51:47.621807 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1cc47a-7b04-40c9-919a-2748ca3de512 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:51:47.680548 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:51:47.681108 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:47.681391 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:47.681667 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] PATH_INFO: `/v3/users/7f7914881e2646229ee619ed5c96666e` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:47.682872 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f7914881e2646229ee619ed5c96666e)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:51:47.684819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:51:47.749776 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f7914881e2646229ee619ed5c96666e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.758324 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f7914881e2646229ee619ed5c96666e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.759147 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f7914881e2646229ee619ed5c96666e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.760084 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f7914881e2646229ee619ed5c96666e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.770096 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-955fd8cd-9a68-4eca-96a0-fe3933a51a97 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f7914881e2646229ee619ed5c96666e'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:47.790922 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1464/2914] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:51:47 2022] DELETE /identity/v3/users/7f7914881e2646229ee619ed5c96666e => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:51:47.971527 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:51:47.973429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:51:47.990429 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed9471a-4c54-41b5-b165-1cdd186a17c8 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:51:48.062827 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:51:48.063581 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:51:48.063844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:51:48.064120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] PATH_INFO: `/v3/projects/3738ca011675489ba88c4265e0f1488b` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:51:48.065983 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3738ca011675489ba88c4265e0f1488b)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:51:48.067941 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:51:48.127974 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3738ca011675489ba88c4265e0f1488b', 'request_id': 'req-55fca6e7-1761-4c14-83f5-26e2fdc65867'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:48.140234 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3738ca011675489ba88c4265e0f1488b', 'request_id': 'req-55fca6e7-1761-4c14-83f5-26e2fdc65867'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:48.149048 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3738ca011675489ba88c4265e0f1488b', 'request_id': 'req-55fca6e7-1761-4c14-83f5-26e2fdc65867'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:48.164840 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] The token cache is being invalidate because project 3738ca011675489ba88c4265e0f1488b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:51:48.165265 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:48.166166 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:51:48.168291 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1451/2915] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:51:47 2022] DELETE /identity/v3/projects/3738ca011675489ba88c4265e0f1488b => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:51:57.786780 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=61,keystone:DELETE=12,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:51:58.159614 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=55,keystone:DELETE=9,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:52:07.565304 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828498e6-b5cf-4f2e-b5c7-9fcf15aca41f None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:52:07.565804 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828498e6-b5cf-4f2e-b5c7-9fcf15aca41f None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:52:07.565921 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828498e6-b5cf-4f2e-b5c7-9fcf15aca41f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:52:07.590796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-828498e6-b5cf-4f2e-b5c7-9fcf15aca41f None None] MFA Rules not processed for user `1addff9d7c7348de827c4eba89d41660`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:52:07.590796 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-828498e6-b5cf-4f2e-b5c7-9fcf15aca41f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:52:07.655447 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1465/2916] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:52:07 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 29 22:52:09.442334 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:52:09.444112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:52:09.519443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fca6e7-1761-4c14-83f5-26e2fdc65867 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:52:09.524417 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:52:09.525024 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:52:09.525300 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:52:09.525567 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:52:09.527869 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:52:09.539934 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:52:09.541525 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:52:09.553058 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1452/2917] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:52:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:52:15.009971 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828498e6-b5cf-4f2e-b5c7-9fcf15aca41f None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:52:15.014785 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-828498e6-b5cf-4f2e-b5c7-9fcf15aca41f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:52:15.110358 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828498e6-b5cf-4f2e-b5c7-9fcf15aca41f None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:52:15.116765 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:52:15.117392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:52:15.117676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:52:15.117951 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:52:15.119762 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:52:15.135230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:52:15.139412 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:52:15.165740 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1466/2918] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:52:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:52:19.548534 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=32 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:52:25.153737 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=59 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:52:25.245845 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:52:25.247338 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:52:25.263879 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a3d4a9-4bcd-4fc9-9d31-8bb7801918cd service placement] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:52:25.270181 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:52:25.270812 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:52:25.271123 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:52:25.271389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:52:25.273157 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:52:25.286357 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:52:25.287988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:52:25.299198 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1453/2919] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:52:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:52:35.294322 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=6 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:53:15.165366 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:53:15.168087 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:53:15.226005 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0633978f-dde8-428b-86a3-5b45c58775e4 service nova] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:53:15.229985 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2daaa28-451e-4f8a-b491-907db653de10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:53:15.230627 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2daaa28-451e-4f8a-b491-907db653de10 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:53:15.230968 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2daaa28-451e-4f8a-b491-907db653de10 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:53:15.231197 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2daaa28-451e-4f8a-b491-907db653de10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:53:15.232932 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-e2daaa28-451e-4f8a-b491-907db653de10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:53:15.244566 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2daaa28-451e-4f8a-b491-907db653de10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:53:15.246576 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2daaa28-451e-4f8a-b491-907db653de10 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:53:15.255542 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1467/2920] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:53:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:53:15.347240 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:53:15.348673 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:53:15.360602 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6132d988-79bb-48d2-af9d-3f22d4cac0ec service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:53:15.364199 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:53:15.364785 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:53:15.365059 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:53:15.365325 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:53:15.367361 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:53:15.381403 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:53:15.382839 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:53:15.393900 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1454/2921] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:53:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:53:25.251532 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=26 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:53:25.388936 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=6 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:54:05.332035 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f714da0d-9fd7-44f6-970e-c9e0c9b7be96 None None] REQUEST_METHOD: `POST` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:54:05.332726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f714da0d-9fd7-44f6-970e-c9e0c9b7be96 None None] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:54:05.332726 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f714da0d-9fd7-44f6-970e-c9e0c9b7be96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:54:05.354247 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.auth.core [None req-f714da0d-9fd7-44f6-970e-c9e0c9b7be96 None None] MFA Rules not processed for user `1addff9d7c7348de827c4eba89d41660`. Rule list: `[]` (Enabled: `True`). {{(pid=103002) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 29 22:54:05.355563 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-f714da0d-9fd7-44f6-970e-c9e0c9b7be96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:54:05.364357 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1468/2922] 10.222.0.77 () {58 vars in 958 bytes} [Mon Aug 29 22:54:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 29 22:54:08.452388 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:54:08.471616 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda653ef-794b-47c2-9c39-2e5c31d49d67 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:54:08.480498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57491e4de6c3435e9d818266a910b831', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service', 'ResellerAdmin', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:54:08.482023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:54:08.482023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:54:08.482023 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:54:08.484112 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:54:08.497121 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:54:08.499347 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:54:08.510293 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1455/2923] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:54:08 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:54:15.347664 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=3 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:54:18.505228 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=6 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:54:45.203838 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f714da0d-9fd7-44f6-970e-c9e0c9b7be96 None None] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:54:45.222364 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f714da0d-9fd7-44f6-970e-c9e0c9b7be96 None None] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:54:45.227761 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:54:45.233164 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:54:45.233468 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:54:45.233751 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:54:45.235819 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:54:45.249565 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:54:45.251203 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:54:45.263148 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1469/2924] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:54:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:54:50.750988 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:54:50.753497 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:54:50.770306 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4979d4-3bda-44a3-a24f-c7a405091635 service nova] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:54:50.776401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:54:50.777164 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:54:50.777561 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:54:50.778010 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:54:50.779950 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:54:50.793681 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:54:50.795337 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:54:50.807389 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1456/2925] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:54:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 29 22:54:54.184665 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:54:54.186166 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:54:54.256154 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbd7c5-c9ab-438c-b4d8-269925e8d9f2 service placement] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:54:54.264065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:54:54.264727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:54:54.265002 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:54:54.265273 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] PATH_INFO: `/v3/users/7447ba5f53114d278711dccc0aa1ad07` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:54:54.266422 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7447ba5f53114d278711dccc0aa1ad07)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:54:54.268972 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:54:54.326027 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7447ba5f53114d278711dccc0aa1ad07'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.333394 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7447ba5f53114d278711dccc0aa1ad07'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.334065 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7447ba5f53114d278711dccc0aa1ad07'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.334669 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7447ba5f53114d278711dccc0aa1ad07'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.341625 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7447ba5f53114d278711dccc0aa1ad07'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.358792 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1470/2926] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:54:54 2022] DELETE /identity/v3/users/7447ba5f53114d278711dccc0aa1ad07 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:54:54.544587 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:54:54.555948 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeb92e7-56c7-4e06-9711-b4bdff49188b service placement] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:54:54.620086 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:54:54.620639 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:54:54.620920 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:54:54.621195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] PATH_INFO: `/v3/projects/01c16f8389774f4c94889a562cdd38f9` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:54:54.622318 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01c16f8389774f4c94889a562cdd38f9)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:54:54.624257 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:54:54.670540 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01c16f8389774f4c94889a562cdd38f9', 'request_id': 'req-59fc6008-0d0f-45d1-95d4-601660a4b3d6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.677443 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01c16f8389774f4c94889a562cdd38f9', 'request_id': 'req-59fc6008-0d0f-45d1-95d4-601660a4b3d6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.684769 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01c16f8389774f4c94889a562cdd38f9', 'request_id': 'req-59fc6008-0d0f-45d1-95d4-601660a4b3d6'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.696366 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] The token cache is being invalidate because project 01c16f8389774f4c94889a562cdd38f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:54:54.696661 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.697320 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.698830 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1457/2927] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:54:54 2022] DELETE /identity/v3/projects/01c16f8389774f4c94889a562cdd38f9 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:54:54.702794 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:54:54.704204 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:54:54.720169 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edff5bfd-0c3e-4f77-a098-7ec4c5452591 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:54:54.777487 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:54:54.778128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:54:54.778404 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:54:54.778676 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] PATH_INFO: `/v3/users/c2db6d92406a45378a34d8b18b3279cb` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:54:54.779848 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2db6d92406a45378a34d8b18b3279cb)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:54:54.781757 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:54:54.787211 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:54:54.843636 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2db6d92406a45378a34d8b18b3279cb'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.854028 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2db6d92406a45378a34d8b18b3279cb'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.854952 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2db6d92406a45378a34d8b18b3279cb'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.855741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2db6d92406a45378a34d8b18b3279cb'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.867741 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2db6d92406a45378a34d8b18b3279cb'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:54.885886 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1471/2928] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:54:54 2022] DELETE /identity/v3/users/c2db6d92406a45378a34d8b18b3279cb => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:54:55.075342 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:54:55.077034 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:54:55.090594 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc6008-0d0f-45d1-95d4-601660a4b3d6 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:54:55.160314 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:54:55.160850 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:54:55.161128 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:54:55.161397 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] PATH_INFO: `/v3/projects/8a4dffe025c1425ea038139156586810` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:54:55.162548 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8a4dffe025c1425ea038139156586810)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:54:55.164483 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:54:55.191982 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:54:55.219764 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8a4dffe025c1425ea038139156586810', 'request_id': 'req-f74de269-548e-4400-945b-2cd06e20e702'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:55.226498 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8a4dffe025c1425ea038139156586810', 'request_id': 'req-f74de269-548e-4400-945b-2cd06e20e702'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:55.242545 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8a4dffe025c1425ea038139156586810', 'request_id': 'req-f74de269-548e-4400-945b-2cd06e20e702'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:55.254409 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] The token cache is being invalidate because project 8a4dffe025c1425ea038139156586810 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:54:55.254687 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:55.255373 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:54:55.256899 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1458/2929] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:54:55 2022] DELETE /identity/v3/projects/8a4dffe025c1425ea038139156586810 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:55:04.882353 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:55:05.251147 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:55:52.482392 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:55:52.484136 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:55:52.555195 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5c481b-d5fc-4444-9a1b-0e14ab618a31 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:55:52.605899 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:55:52.606547 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:55:52.606871 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:55:52.607142 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:55:52.608926 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:55:52.692450 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:55:52.693902 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:55:52.710605 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1472/2930] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:55:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:55:53.900405 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:55:53.901928 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:55:53.914177 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74de269-548e-4400-945b-2cd06e20e702 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:55:53.917937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:55:53.918535 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:55:53.918844 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:55:53.919116 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:55:53.920890 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:55:53.936619 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:55:53.938217 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:55:53.951012 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1459/2931] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:55:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:56:02.701272 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=74 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:56:03.945926 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=6 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:56:04.865976 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:56:04.874948 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55719d44-708c-4d9c-aaec-1106663f7169 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:56:04.878590 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dea13198ef8e45de85b92e79e3ee71b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:56:04.879226 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:56:04.879506 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:56:04.879777 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:56:04.881557 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:56:04.945771 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:56:04.947155 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:56:04.957134 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1473/2932] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:56:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:56:11.192575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:56:11.194202 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:56:11.268189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:56:11.268189 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:56:11.281415 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c4e022-ded6-4a5f-a23b-b632b4027c27 service neutron] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:56:11.286610 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:56:11.287923 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:56:11.288217 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:56:11.288493 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] PATH_INFO: `/v3/users/1addff9d7c7348de827c4eba89d41660` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:56:11.289728 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1addff9d7c7348de827c4eba89d41660)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:56:11.291826 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:56:11.300738 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d67642-5656-4f36-887e-a103445cbe22 service neutron] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:56:11.305933 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:56:11.306542 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] REQUEST_METHOD: `GET` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:56:11.306849 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:56:11.307120 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:56:11.308865 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:56:11.322636 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:56:11.324625 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:56:11.335832 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1460/2933] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 22:56:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 22:56:11.347110 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1addff9d7c7348de827c4eba89d41660'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.355255 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1addff9d7c7348de827c4eba89d41660'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.355517 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1addff9d7c7348de827c4eba89d41660'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.356128 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1addff9d7c7348de827c4eba89d41660'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.367258 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1addff9d7c7348de827c4eba89d41660'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.388727 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1474/2934] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:56:11 2022] DELETE /identity/v3/users/1addff9d7c7348de827c4eba89d41660 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:56:11.569710 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:56:11.582575 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155c2cfe-9398-4832-a30b-ab44ffa54d9a service placement] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:56:11.640927 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:56:11.641549 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:56:11.641823 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:56:11.642092 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] PATH_INFO: `/v3/projects/36e090f60dd244cea327de965ad9af13` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:56:11.643789 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36e090f60dd244cea327de965ad9af13)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:56:11.645829 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:56:11.696055 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36e090f60dd244cea327de965ad9af13', 'request_id': 'req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.702453 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36e090f60dd244cea327de965ad9af13', 'request_id': 'req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.711668 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36e090f60dd244cea327de965ad9af13', 'request_id': 'req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.724195 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] The token cache is being invalidate because project 36e090f60dd244cea327de965ad9af13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:56:11.724487 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.725160 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.727045 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1461/2935] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:56:11 2022] DELETE /identity/v3/projects/36e090f60dd244cea327de965ad9af13 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 29 22:56:11.730842 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:56:11.732354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:56:11.746012 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745e93f-6360-47dc-a998-dc3fc198922a admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:56:11.810551 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:56:11.811354 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:56:11.811634 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:56:11.811907 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] PATH_INFO: `/v3/users/576a13523a5d4cbba20fbd95c183a173` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:56:11.813043 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=576a13523a5d4cbba20fbd95c183a173)` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:56:11.815626 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:56:11.821586 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:56:11.867712 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '576a13523a5d4cbba20fbd95c183a173'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.880708 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '576a13523a5d4cbba20fbd95c183a173'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.881801 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '576a13523a5d4cbba20fbd95c183a173'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.882849 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '576a13523a5d4cbba20fbd95c183a173'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.893585 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.notifications [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '576a13523a5d4cbba20fbd95c183a173'} {{(pid=103002) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:11.923335 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1475/2936] 10.222.0.77 () {58 vars in 1224 bytes} [Mon Aug 29 22:56:11 2022] DELETE /identity/v3/users/576a13523a5d4cbba20fbd95c183a173 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:56:12.099959 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] Authenticating user token {{(pid=103003) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 22:56:12.101448 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.fernet_utils [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103003) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 22:56:12.117052 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf9257f-9c6a-4dda-8f3e-e66a30fa3eb3 admin admin] Validating token access rules against request {{(pid=103003) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 22:56:12.135650 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:56:12.178583 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f1e799f7d5b4182a2e0ecd338aaa415', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fabab14479bd4d4c870f8f76efa1b499', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103003) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 22:56:12.179251 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] REQUEST_METHOD: `DELETE` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 22:56:12.179530 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] SCRIPT_NAME: `/identity` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 22:56:12.179803 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] PATH_INFO: `/v3/projects/7e728c6888fe4d159900e8212c6c2e81` {{(pid=103003) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 22:56:12.180929 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e728c6888fe4d159900e8212c6c2e81)` {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 22:56:12.182867 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] RBAC: Authorization granted {{(pid=103003) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 22:56:12.234411 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e728c6888fe4d159900e8212c6c2e81', 'request_id': 'req-d0a26d7b-8541-4c52-b2b6-70f420934310'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:12.241266 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e728c6888fe4d159900e8212c6c2e81', 'request_id': 'req-d0a26d7b-8541-4c52-b2b6-70f420934310'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:12.249005 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e728c6888fe4d159900e8212c6c2e81', 'request_id': 'req-d0a26d7b-8541-4c52-b2b6-70f420934310'} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:12.260937 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] The token cache is being invalidate because project 7e728c6888fe4d159900e8212c6c2e81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=103003) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 29 22:56:12.261398 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:12.261873 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG keystone.notifications [None req-d0a26d7b-8541-4c52-b2b6-70f420934310 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=103003) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 29 22:56:12.263401 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: [pid: 103003|app: 0|req: 1462/2937] 10.222.0.77 () {58 vars in 1233 bytes} [Mon Aug 29 22:56:12 2022] DELETE /identity/v3/projects/7e728c6888fe4d159900e8212c6c2e81 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 29 22:56:21.916735 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 22:56:22.257068 dv-cinder-854270-4-iscsi devstack@keystone.service[103003]: DEBUG dbcounter [-] [103003] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=103003) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 29 23:00:12.718322 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] Authenticating user token {{(pid=103002) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 29 23:00:12.719768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 23:00:12.866230 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf0162-5309-4f00-8522-1a79fabe3d72 admin admin] Validating token access rules against request {{(pid=103002) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 29 23:00:12.928384 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43afb849-e7c3-4127-8c18-299dadb2246f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b6f80303c62847ca9a506d58d9b6f396', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3d28d10d57ea42408a6ced32a39139c9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=103002) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 29 23:00:12.928995 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afb849-e7c3-4127-8c18-299dadb2246f service placement] REQUEST_METHOD: `GET` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 29 23:00:12.929271 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afb849-e7c3-4127-8c18-299dadb2246f service placement] SCRIPT_NAME: `/identity` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 29 23:00:12.929539 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43afb849-e7c3-4127-8c18-299dadb2246f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=103002) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 29 23:00:12.931845 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.fernet_utils [None req-43afb849-e7c3-4127-8c18-299dadb2246f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=103002) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 29 23:00:12.943768 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43afb849-e7c3-4127-8c18-299dadb2246f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 29 23:00:12.945346 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43afb849-e7c3-4127-8c18-299dadb2246f service placement] RBAC: Authorization granted {{(pid=103002) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 29 23:00:12.956150 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: [pid: 103002|app: 0|req: 1476/2938] 10.222.0.77 () {60 vars in 1365 bytes} [Mon Aug 29 23:00:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 29 23:00:22.951791 dv-cinder-854270-4-iscsi devstack@keystone.service[103002]: DEBUG dbcounter [-] [103002] Writing DB stats keystone:SELECT=89 {{(pid=103002) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}